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Background and Study Overview 


Uncommon Schools is a nonprofit charter management organization that starts and manages charter 
schools. The organization’s approach is well established; Uncommon opened its first school in New 
Jersey more than 20 years ago. As of 2021, the organization operates 55 schools across Boston 
(Massachusetts), Camden (New Jersey), New York City, Newark (New Jersey), Rochester (New York), 
and Troy (New York); 36 percent are elementary schools, 45 percent are middle schools, and 18 percent 
are high schools. Historically, schools in the Uncommon network have focused on six key components: 


1. A college preparatory mission, which is infused throughout the school environment at all grade levels 


2. High standards for academics and character, including a rigorous curriculum and focus on student 
achievement 


3. A highly structured and joyful learning environment, in which teachers are trained and supported to 
maximize instructional time 


4. A longer school day and a longer school year than for typical schools 


A focus on accountability and data-driven instruction, with school leaders using data on student 
progress to inform instructional changes 


6. A faculty of committed, talented, and well-trained teachers and leaders, emphasizing commitment to 
the Uncommon mission and frequent opportunities for teachers to develop through trainings, 
observations, and feedback 


Uncommon’s model for operating schools is promising, with rigorous evidence of effectiveness in 
improving student outcomes. Several previous studies of Uncommon schools found that enrollment in an 
Uncommon school leads to statistically significant and positive impacts on student achievement 
(Furgeson et al. 2012; Teh et al. 2010; Woodworth and Raymond 2013; Woodworth et al. 2017). One of 
the studies was reviewed and met What Works Clearinghouse evidence standards with reservations 
(Furgeson et al. 2012). The studies were limited to Uncommon schools located in Newark, New York 
City, and Rochester. Two studies examined students in middle school (Furgeson et al. 2012; Teh et al. 
2010), and two studies examined students across all grade levels (Woodworth and Raymond 2013; 
Woodworth et al. 2017). 


Uncommon typically opens new schools, building up from the earliest grades. However, as part of 
broader urban renewal initiatives and school turnaround efforts in Camden and Newark, New Jersey, 
Uncommon recently sought to apply its school model in a turnaround setting. The organization 
collaborated with two public school districts, Camden City School District and Newark Public Schools, to 
implement the Uncommon school model as a strategy for turning around persistently low-performing 
schools. In the Camden City School District, Uncommon partnered with the district as part of a 
renaissance school project, which allows a nonprofit entity to operate and manage a group of new or 
existing schools that are part of the district in an urban campus area. ' In 2015-2016, Uncommon and the 
Camden City School District launched their first turnaround of a low-performing school (Bonsall 
Elementary School) in grades 1 through 4.7 Uncommon’s partnership with Newark Public Schools was 


' Urban Hope Act, N.J.S.A. 18A:36C-13. 
? During the prior school year, 2014-2015, Uncommon launched a limited turnaround effort, assuming management 
only of the school’s kindergarten. 
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focused on a single charter restart of a struggling school. Uncommon took over the management of one of 
New Jersey’s lowest-performing schools, Alexander Street Elementary, in 2014-2015. 


Uncommon was awarded a 2016 grant from the U.S. Department of Education’s Investing in Innovation 
Fund (13) to support TurnNJ, a project intended to scale the initial implementation work of Uncommon’s 
whole-school turnaround strategy in Alexander Street Elementary School and Camden Prep Mt. Ephraim 
Elementary School to additional grade levels in the two schools and to four other low-performing schools 
across Camden and Newark. The project continued Uncommon’s existing partnerships with the Camden 
City School District and Newark Public Schools. In Camden, Uncommon now manages two elementary 
schools, one middle school, and one high school that remain part of Camden City School District; a 
second middle school will open in 2021—2022. In Newark, Uncommon continues to manage the same 
elementary school. (The New Jersey Department of Education is not involved in TurnNJ.) 


Although there is promising evidence of the effectiveness of the Uncommon school model in a new and 
continuing charter school setting, the approach is untested in a struggling neighborhood school. The 
TurnNJ project hypothesizes that the Uncommon school model will lead to success when implemented in 
a turnaround setting (Figure 1). TurnNJ schools operate with the same six key attributes as the 
Uncommon Schools network, described above. Together, these six attributes are meant to improve school 
quality, student attendance, and student achievement in the short term, which will in turn lead to increased 
high school graduation, postsecondary attainment, and earnings, as well as reduced criminality. 


—_ ey 
Figure 1. Uncommon TurnNJ’s logic model 


TurnNJ model Intermediate outcomes Long-term outcomes 


* Underserved students * College preparatory mission * Improved school quality + Improved high school 
+ Family, community, policy + High standards + Improved student graduation rates 
supports * Structured learning attendance * Increased postsecondary 


* Talent pipeline » Longer school day and year * Improved student attainment 
achievement * Higher earnings 


* Funding * Data-driven instruction 
O © 6, © + Reduced criminality 


* Talented educators 
+ Replication of 


TurnNJ model 


To assess the effectiveness of the TurnNJ project, as required by its i3 grant, Uncommon engaged 
Mathematica as an independent evaluator. The goal of the study is to rigorously estimate the impact 
TurnNJ schools have on the math and English language arts (ELA) achievement of upper elementary and 
middle school students. Specifically, the study addresses the following research questions: 


e What is the impact of enrollment in a TurnNJ school on students’ math achievement one, two, three, 
and four years after enrollment? 


e What is the impact of enrollment in a TurnNJ school on students’ ELA achievement one, two, three, 
and four years after enrollment? 


In addition, the study investigated the extent to which the TurnNJ schools implemented the six 
components of the Uncommon school model as intended. 


Two key findings emerged: 
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e Enrollment in a TurnNJ school had positive and statistically significant impacts on student 
achievement in math that persisted up to four years after enrollment. After one year of enrollment, 
TurnNJ students had test score gains equal to an additional 1.8 years of learning in math compared to 
similar students at other schools. TurnNJ’s cumulative impact after four years of enrollment exceeded 
the size of the poverty test score gap in math. 


e Enrollment in a TurnNJ school had positive and statistically significant impacts on student 
achievement in English language arts that persisted up to four years after enrollment. After one year 
of enrollment, TurnNJ students had test score gains equal to an additional 1.9 years of learning in 
ELA relative to similar students at other schools. TurnNJ’s cumulative impact after four years of 
enrollment was about 89 percent of the poverty test score gap in ELA. 


These findings contribute to the existing literature on the effectiveness of the Uncommon school model 
and provide new evidence on the implementation of the Uncommon school model in a turnaround setting. 
Mixed evidence from previous studies examining the effectiveness of other turnaround initiatives 
suggests that designing and implementing a turnaround strategy that can improve student achievement is 
challenging (Redding and Nguyen 2020). The initial success of the TurnNJ strategy in a limited number 
of schools may offer a starting point for expanding the approach to turning around low-performing 
schools on a larger scale. 
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Sample and Methods 


To address the study research questions, we first identified a comparison group of students who never 
attended a TurnNJ school but were similar to students who enrolled at TurnNJ schools. We then 
compared the achievement outcomes for TurnNJ students to those for the comparison group by using a 
regression model that accounted for students’ demographic characteristics and math and ELA 
achievement before enrolling in a TurnNJ school. In this chapter, we provide a brief overview of our 
approach. A more detailed description of the methods is provided in Appendix A. 


Sample 


Selecting TurnNJ schools 


We sought to include as many TurnNJ schools in the sample as possible so that the results from the study 
would apply to a broad range of schools and grade levels. Four TurnNJ schools opened within the time 
frame of the study, and our sample includes three of the four schools: Alexander St. Elementary School 
(ASES), which reopened in Newark in 2014—2015 under the charter restart model; Camden Prep Mt. 
Ephraim Elementary School (CPMES), which reopened in Camden in 2015-2016 as a renaissance 
school; and Camden Prep Mt. Ephraim Middle School (CPMMS), which opened as a new renaissance 
school in 2016-2017. We could not include one school because it started with students in earlier grade 
levels who did not have the required data for analysis.* The results from the analysis therefore pertain to 
these three TurnNJ schools during this time period but may be suggestive of results one might expect 
from similar turnaround schools in urban settings. 


Identifying students in TurnNJ schools 


Within the three study schools, we tracked data on groups of students who began school at the same 
time—treferred to hereafter as cohorts—to assess their math and ELA achievement for up to four years 
following their initial enrollment (Figure 2). The two elementary schools in the study are represented by 
one cohort of students who enrolled in the TurnNJ school starting in 4th grade. We limited the sample to 
students who were in 4th grade during the opening year because that was the only grade and year in 
which students had test scores before attending a TurnNJ school (as required for our analyses). In both 
schools, approximately half of students in the sample previously attended the low-performing 
neighborhood school in 3rd grade, before it reopened under Uncommon management; the other half came 
from other schools. 


3 A second elementary school in Camden was not included because it opened with kindergarten and Ist grade. 
Students who began at a school before 4th grade could not be included because they did not have test scores the year 
prior to enrollment, which were needed for the analysis. A TurnNJ high school opened after the study period. 


Mathematica 


Impacts of Uncommon Schools in a Turnaround Setting 


Figure 2. Timeline of school openings and cohorts 


ASES opens CPMES opens CPMMS opens 
2014-2015 2015-2016 2016-2017 2017-2018 2018-2019 


Ns) =to mee) a10) it @) ed \V/| tooo) ale) at (@) =) V/V ks iexe) ale) a am CPMMS cohort 2 CPMMS cohort 3 
(CofgeTe(=a-9) (grade 4) (grade 5) (grades 5/6) (grades 5/6) 


Note: ASES = Alexander St. Elementary School; CPMES = Camden Prep Mt. Ephraim Elementary School; CPMMS 
= Camden Prep Mt. Ephraim Middle School. 


CPMMS includes three consecutive cohorts of students. The school was built up from 5th grade, so the 
first cohort includes only incoming 5th graders, and the subsequent cohorts include incoming 5th and 6th 
graders who were enrolling in an Uncommon school for the first time.* 


Within the school cohorts, we identified and included in the sample TurnNJ students who had never 
previously attended an Uncommon school, had math and ELA test scores from the prior school year, and 
had no missing data on demographic characteristics. For the middle school, we excluded students from 
the sample who had previously attended an Uncommon elementary school, in order to isolate the impact 
of enrolling in a TurnNJ school.° Students who attended an Uncommon elementary school did not have 
math and ELA test scores before enrolling in an Uncommon school. Therefore, we would not have been 
able to capture the full, cumulative impact of new exposure to the Uncommon school model in the 
TurnNJ setting. Finally, we removed students if we could not identify any students from other schools 
with similar traits during our matching process, which we describe in the next section. Figure 3 illustrates 
this process for defining an analysis sample, using the students included in our analysis of one-year math 
outcomes as an example. (Sample sizes for all analysis samples are included in Tables A.2 and A.3 in 
Appendix A.) 


4 We did not include incoming 7th graders because there are very few students coming into a new middle school in 
7th grade. 

5 To examine the impacts of TurnNJ for as expansive a sample as possible, we conducted a separate analysis that 
included both TurnNJ students new to Uncommon and TurnNJ students continuing from other Uncommon schools; 
findings from this analysis are presented in Appendix C. 
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Figure 3. Flow of TurnNJ students from initial enrollment to analysis sample (one-year math 
impact sample) 


Students enroll in Students have Students are 
TurnNJ school required baseline matched to at 


in fall of Year 1 and outcomes least one 
data for analysis comparison student 


Total: 159 Total: 91 Total: 90 
ASES: 30, CPMES: 43, ASES: 13, CPMES: 29, ASES: 13, CPMES: 28, 
CPMMS: 86 CPMMS: 49 CPMMS: 49 


Source: Administrative data from the New Jersey Department of Education from the 2013-2014 to 2018-2019 
school years. 


Note: ASES = Alexander St. Elementary School; CPMES = Camden Prep Mt. Ephraim Elementary School; 
CPMMS = Camden Prep Mt. Ephraim Middle School. The number of students shown in the first box 
excludes those who had previously attended an Uncommon school. 


These are the sample sizes for the one-year math outcome. The other outcomes have different sample 
sizes and are reported in Appendix A. 


The sample of students from TurnNJ schools included in this study appeared to have demographic 
characteristics similar to those of the broader Uncommon network. Both the Uncommon network and the 
subset of Uncommon TurnNJ schools in our sample serve a large percentage of students from 
traditionally disadvantaged groups, including families with low incomes. For example, in its 13 
application, Uncommon reported that 83 percent of students in its network come from families with low 
incomes; our samples included between 91 and 97 percent of children from families with low incomes 
based on eligibility for free or reduced-price lunch. 


Identifying similar students in other schools 


The primary goal of this evaluation is to compare the level of achievement in math and ELA of TumNJ 
students to their expected level of achievement if they had enrolled in other public schools in their cities. 
We refer to the difference between students’ level of achievement in TurnNJ and their expected 
achievement if enrolled at other schools as an “impact estimate.” Because it is not possible for any student 
to attend a TurnNJ school and a non-TurnNJ school at the same time, the best way to calculate this 
estimate is to compare the TurnNJ students to students who are very similar but attended other local 
schools. Our impact estimates are more likely to be accurate when the groups of students are more alike. 


Randomly assigning students to attend schools would ensure that the TurnNJ and comparison students are 
alike, which is why random assignment studies are known as the “gold standard” in research. When 
students are randomly assigned to schools, we can assume that all of the characteristics of students in 
those schools—including characteristics that can be measured, such as prior ELA and math achievement, 
and characteristics that cannot be measured, such as parent engagement—are similar. We can then 
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estimate impacts simply by comparing the means in math and ELA between students in the two schools 
without concern that non-school factors might be contributing to differences in achievement. 


However, when students and their families choose to attend certain schools and are not randomly assigned 
to them, as is the case with TurnNJ schools, the factors that influence their decision to enroll in a school 
often influence their achievement. For example, parents who prefer a school with a record of high student 
achievement may be more likely to enroll their child in a particular school and more likely to engage in 
other activities, such as providing tutoring or help with homework, that influence their child’s 
achievement. Because of these selection biases, we cannot simply calculate the impact estimate as the 
difference between all students’ achievement levels in the different schools. This estimate would capture 
both the differences between the schools and the differences between the types of students and families 
that attend them. Instead, we must identify a subset of non-TumnNJ students who are most similar to 
TumNJ students in meaningful ways and adjust for any remaining differences between them to minimize 
selection biases. 


To respond to this challenge, we matched each student who attended a TurnNJ school with up to 10 
students who were very similar but attended a different public school in the same city, using a method 
called propensity score matching. We matched students based on their standardized test scores from the 
prior school year and demographic characteristics shown in Box 1, separately for each cohort and for each 
of the eight outcomes: math achievement one, two, three, and four years after enrollment and ELA 
achievement one, two, three, and four years after enrollment. 


After matching, we confirmed that the TurnNJ students and the comparison students who were matched 
to them had similar prior test scores in math and ELA. Across the four years, the samples used to evaluate 
math achievement had differences in prior math scores between 0.05 and 0.12 standard deviations, and all 
of the samples used to evaluate ELA achievement had differences in baseline ELA scores between 0.00 
and 0.02 (see tables of baseline math and ELA scores in Appendix A). These fall well below the 
equivalence threshold of less than 0.25 standard deviations set by the What Works Clearinghouse, the 
body funded by the U.S. Department of Education that evaluates the rigor of educational research studies 
funded by the U.S. Department of Education. 


Box 1. Data used in the analysis 
e Source: New Jersey Department of Education 
e Data used for matching: 


— Student demographics, including gender, race, ethnicity, English learner status, special 
education status, low-income status, attendance, grade level 


— Standardized test scores in math and ELA from the prior school year 
— Information on testing accommodations and alternative assessments 


e Data used to assess outcomes: standardized test scores in math and ELA from one to four years 
following enrollment in TurnNJ 


e Grades: Pre-K through 8 
e School years: 2013-2014 through 2018-2019 


e Schools: All public schools in the cities of Camden and Newark, including district, renaissance, and 
charter schools 
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We also confirmed that the TurnNJ students and matched students had similar demographic 
characteristics. Figure 4 shows three groups of students: the green bar represents the students included in 
the analysis who attended TurnNJ schools; the gold bar represents the students who were matched to 
TurnNJ students and thus included in the analyses; and the navy blue bar represents the comparison 
students before matching (the complete pool of students in Newark and Camden in the same grade levels 
and years who did not enroll in any Uncommon school). As the figure illustrates, the matched group is 
much more similar to the TurnNJ students than the larger pool of students in those cities. 


Figure 4. Baseline characteristics of TurnNJ students and comparison students before and after 
matching (one-year math impact sample) 
100 


90 


80 


Percentage at baseline 


Female Hispanic Black FRPL Special Chronic 
education absentee 


[| TurnNJ students ma Comparison students: After matching | Comparison students: Before matching 
Source: Administrative data from the New Jersey Department of Education from the 2013-2014 to 2018-2019 
school years. 


Note: Chronic absentee is defined as a student missing more than 10 percent of total school days in any baseline 
year. FRPL, special education, and EL status were determined based on whether a student ever had such 
a status in any baseline year. 


EL = English learner; FRPL = eligible for free or reduced-price lunch. 


Outcomes 


We measured math and ELA achievement using standardized test scores one to four years following 
enrollment in a TurnNJ school. We converted the scale scores to z-scores; within each city, school year, 
and grade level, we rescaled the scores to have a mean of 0 and standard deviation of 1. Therefore, 
negative scores indicate scores below the mean of students in the same city, school year, and grade level, 
and positive scores indicate those above the mean. 
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We pooled, or combined data, across schools, cohorts, and school years to analyze outcomes for one year, 
two years, three years, and four years following enrollment. Figure 5 presents a timeline for the 
evaluation, including when each school opened, when students from each cohort enrolled in a TumNJ 
school, and when we collected outcomes data for each cohort. As shown in the figure, each cohort started 
in a different year between 2014-2015 and 2018-2019, and as a result, their outcomes from Years 1-4 
were collected in different years. The cohorts contributing to outcomes in each year after enrollment are 
shown in one color along a diagonal. For example, the Year 1 outcomes, which are shaded in red, consist 
of the 4th grade cohort from ASES collected in 2014-2015, the 4th grade cohort from CPMES collected 
in 2015-2016, the Sth grade cohort from CPMMS collected in 2016-2017, and the 5th and 6th grade 
cohorts from CPMMS collected in 2017-2018 and 2018-2019. Because 2018-2019 was the last school 
year of available data and the CPMMS cohorts began later, one CPMMS cohort drops out of the analysis 
each year beginning in Year 2. (The Year 1 outcomes have three CPMMS cohorts, the Year 2 outcomes 
have two CPMMS cohorts, the Year 3 outcomes have one CPMMS cohort, and the Year 4 outcomes have 
no cohorts.) We treat the student as the unit of analysis. Therefore, all students have equal weight in each 
analysis even though the schools and cohorts do not. 


Figure 5. Timeline of school openings, cohort entry, and outcome measurement 


ASES opens CPMES opens CPMMS opens 
2014-2015 2015-2016 2016-2017 2017-2018 2018-2019 
Oe \V/| =k iexe) ae) a CPMMS cohort 1 CPMMS cohort 2 #4 CPMMS cohort 3 


ASES cohort 


(Cole=tol=a-9) (grade 4) (Cofeete (=e) (ol r=To[=tsey))) ((ole=To[=\sireyLo)) 
ASES cohort ASES cohort ASES cohort ASES cohort 
Year 1 outcomes Year 2 outcomes Year 3 outcomes Year 4 outcomes 

CPMES cohort CPMES cohort CPMES cohort CPMES cohort 

Year 1 outcomes Year 2 outcomes Year 3 outcomes Year 4 outcomes 

CPMMS cohort 1 CPMMS cohort 1 CPMMS cohort 1 

Year 1 outcomes Year 2 outcomes Year 3 outcomes 

CPMMS cohort 2 CPMMS cohort 2 

Year 1 outcomes Year 2 outcomes 

CPMMS cohort 3 


Year 1 outcomes 


Note: ASES = Alexander St. Elementary School; CPMES = Camden Prep Mt. Ephraim Elementary School; 
CPMMS = Camden Prep Mt. Ephraim Middle School. Cohorts with the same color are those that are 
included in the analysis of an outcome. For example, the Year 4 outcomes include the ASES 4th grade 
cohort in 2017-2018 and the CPMES 4th grade cohort in 2018-2019. 
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A complication with analyzing outcomes across four years is that not all students stay in the same schools 
during these years. In the case of the elementary cohorts from ASES and CPMES, all students attended 
other schools in Year 2 because their cohorts began in 4th grade, which is the last grade level in these 
elementary schools. In most cases, the middle school attended was also an Uncommon school: 87 percent 
of students in the ASES cohort attended other Uncommon middle schools in Newark that were not part of 
TumNJ, and 77 percent of students in the CPMES cohort attended CPMMS. Therefore, in the case of the 
elementary cohorts, the outcomes in Years 2-4 largely capture a mix of the impact of attending the 
TurnNJ elementary school in Year | and the impact of also attending CPMMS or another Uncommon 
middle school in subsequent years. 


We also included students in the analysis who left the Uncommon network at any point during their 
trajectory. This classification enabled us to account for possible bias from certain types of students 
leaving TurnNJ schools (for example, if students who are struggling academically are more likely to 
leave). However, because we did not collect data outside of public schools in Newark and Camden, we 
were not able to follow students who moved outside of those cities. Seventeen percent of TumnNJ students 
in our Year | impact sample were not included in the impact estimates for Year 2 and beyond because 
they moved to another city. 


Primary approach for estimating impacts 


Because students were not randomly assigned to schools, we used statistical methods to account for 
differences between the TurnNJ and comparison students in our analysis of impacts. Our propensity score 
matching approach resulted in TurnNJ and comparison students with similar prior math and ELA 
achievement and demographic characteristics. To account for any remaining differences on these 
observable characteristics, we estimated impacts using a regression model that controlled for math and 
ELA scores from the prior school year and the same demographic characteristics used for propensity 
score matching. We conducted the analysis separately for each combination of subject and number of 
years after enrollment. The full regression model and details about the construction of the variables 
included in the propensity score models and impact models are provided in Appendix A. 


Results of these analyses should remove much of the bias in the impact estimates if the propensity score 
model and impact model are able to account for all factors that influence both the likelihood that a student 
would enroll in a TurnNJ school and math and ELA achievement. Our study accounted for differences on 
observable student characteristics, including prior standardized test scores and demographic 
characteristics, using propensity score matching and regression analyses. However, there may be 
unobservable characteristics that influence both enrollment in TurnNJ and academic achievement that our 
statistical methods do not account for, which would bias our results. For example, the study did not 
account for internal traits, such as motivation, or characteristics of students’ families, neighborhoods, and 
the prior schools they attended. If any of these factors influenced a student’s decision to enroll and 
influenced math or ELA achievement, the results of the study would not be equivalent to the results from 
a randomized controlled trial. This is a limitation of any study that does not use random assignment and 
compares the outcomes of students from different schools. 
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Alternative approaches for estimating impacts 


When estimating impacts, it is important to 


understand how consistent the findings are when Box 2. Types of alternative approaches 
using different analytic approaches (Box 2). For tested 

example, a key challenge in this evaluation is that e Different criteria for selecting students. 
different cohorts and schools contributed to each For example, one approach limited 

outcome year, and we responded by giving each comparison students to those from the same 
student in the analysis equal weight, even if it meant feeder schools as the TurnNJ students. 


that in some years we had greater representation from 
certain schools than in other years. But if one of the 
schools had a larger share of students and had larger 
impacts than other schools, then the decision to 
weight the analysis equally across schools would 
result in smaller impact estimates than if weighted 
equally across students. To address this concern, we 
also conducted one version of the analysis that gave 


Different propensity score methods. For 
example, we tested using the propensity 
scores for weighting rather than matching 
and tested different matching algorithms. 


e Different impact model. We tested one 
variation that had a simplified regression 
model with fewer variables. 


equal weights to each school in each outcome year. e Different weighting. We tested weighting 
By estimating the impacts using a variety of the impacts equally by school and weighting 
approaches, we could determine whether any equally by cohort. 


alternative choices would have made a meaningful 

difference in our results. If the results are similar across many alternative approaches, it gives us greater 
confidence that our results accurately reflect the effect of TurnNJ schools on student achievement. We 
used a total of nine alternative approaches to the analysis, and we summarize the results in the next 
section. (A detailed description of the types of approaches we tested is in Appendix C.) 
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Results 


Students who enrolled in a TurnNJ school had higher average achievement in math and ELA compared to 
similar students who attended other schools. The benefits for students persisted for all four years after 
their initial enrollment in a TurnNJ school. 


Math impacts 


The impacts of enrolling in a TurnNJ school on math achievement were large and statistically significant 
for one, two, three, and four years after enrollment. Figure 6 shows the mean math scores for the TurnNJ 
students in green and for the comparison students in gold. The brackets show that the impact estimate is 
the difference between these means. In the year before enrollment, the average math achievement of these 
groups was essentially the same; TurnNJ students and their matched peers had average z-scores of -0.39 
and -0.35, respectively. But only one year after enrollment, TurnNJ students went from being well below 
their city averages to well above them, moving from a z-score of -0.39 to 0.62, whereas the matched 
students had little change in their standing relative to the city means. 


Students who enrolled in TurnNJ schools maintained their higher scores in the three years that followed. 
By four years after enrollment, math achievement outcomes for students who enrolled in a TumnNJ school 
exceeded those of their matched counterparts by nearly 1 standard deviation. One caveat is that the four- 
year outcomes are based solely on the students who began in 4th grade at the two elementary schools; it is 
unclear whether we can expect these large impacts to persist for students who were only followed in the 
analysis for one or two years. 


These effect sizes translate to large differences in math learning between TurnNJ students and their most 
similar peers. Using benchmarks by Bloom et al. (2008) on average learning gains, TurnNJ students were 
ahead of their matched peers by approximately 1.8 additional years, or 22 additional months, of learning 
in math after one year of enrollment.° By four years, students who enrolled in a TurnNJ school were 
ahead of their peers by approximately 2.1 additional years, or 26 additional months, of learning in math. 


Given that the vast majority (97 percent) of TurnNJ students are eligible for free or reduced-price lunch, 
another benchmark for interpreting TurnNJ’s impacts on student achievement is the achievement gap in 
8th grade between those who do and do not live in poverty. On average, 8th-grade students who are not 
eligible for free or reduced-price lunch perform 0.8 standard deviations in math above those who are 
eligible (Bloom et al. 2008). Uncommon’s impact on math achievement four years after enrollment 
exceeds the size of the gap between students who are eligible for free or reduced-price lunch and those 
who are not. (See Appendix B for supplemental tables showing the sample sizes, impact estimates, 
standard errors, and p-values for each outcome subject and year.) 


® See Appendix A for detailed information on the conversion of effect sizes to years of learning. 
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Figure 6. TurnNJ impacts on math achievement, by years after enrollment 
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Average math z-scores 


1 year after enrollment 2 years after enrollment 3 years after enrollment 4 years after enrollment 


[] TurnNJ students a Matched comparison students 


Source: Administrative data from the New Jersey Department of Education from the 2013-2014 to 2018-2019 
school years. 


Note: This figure reports the regression-adjusted means in math achievement for students who first enrolled in a 
TurnNJ school in 4th, 5th, or 6th grade and the unadjusted means for matched students in Camden and 
Newark who never enrolled in a TurnNJ school or in any other Uncommon school. The means are reported 
in standard deviation units so that the difference between the green bar and the gold bar represents the 
estimated effect size. (Differences may deviate from reported impact sizes in Table A.15 by one-tenth due 
to rounding.) Standardized test scores were normalized to have a mean of 0 and standard deviation of 1, by 
grade level, city (Camden or Newark), and school year. 


* Significantly different from zero at the .05 level, two-tailed test. 
** Significantly different from zero at the .01 level, two-tailed test. 


ELA impacts 


Students who enrolled at a TumnNJ school also saw large gains in ELA achievement compared to their 
matched peers. The impacts were large and statistically significant at one, two, three, and four years 
following enrollment (Figure 7). In the year before enrollment, TurnNJ and comparison students both had 
average scores well below the city averages with z-scores of -0.27 and -0.23, respectively. After one year 
of enrollment in a TurnNJ school, students’ average ELA scores improved to 0.46 standard deviations 
above the city mean, whereas the matched comparison students had nearly the same average z-score as 
before. As a result, the TurnNJ students scored 0.70 standard deviations higher than the matched 
comparison students. These gains were equivalent to approximately 1.8 additional years, or 22 additional 
months, of learning in ELA. 
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Figure 7. TurnNJ impacts on English language arts achievement, by years after enrollment 
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[] TurnNJ students a Matched comparison students 


Source: Administrative data from the New Jersey Department of Education from the 2013-2014 to 2018-2019 
school years. 


Note: This figure reports the regression-adjusted means in English language arts achievement for students who 
newly enrolled in a TurnNJ school in 4th, 5th, or 6th grade and the unadjusted means for matched students 
in Camden and Newark who never enrolled in a TurnNJ school or in any other Uncommon school. The 
means are reported in standard deviation units so that the difference between the green bar and the gold 
bar represents the estimated effect size. Standardized test scores were normalized to have a mean of 0 
and standard deviation of 1, by grade level, city (Camden or Newark), and school year. 


* Significantly different from zero at the .05 level, two-tailed test. 
** Significantly different from zero at the .01 level, two-tailed test. 


The impacts largely persisted for up to four years after enrollment, although they were slightly smaller in 
the fourth year relative to the first three years after enrollment. However, the four-year outcomes did not 
include any of the cohorts that started in middle school, so it is unclear whether the trend would be the 
same with the middle school cohorts. The four-year impacts were still large and statistically significant. 


The gains in ELA achievement were also substantial when comparing them to the achievement gap in 8th 
grade between students who do and do not live in poverty. On average, 8th-grade students who are not 
eligible for free or reduced-price lunch perform 0.66 standard deviations in ELA above those who are 
eligible (Bloom et al. 2008). Uncommon’s impact on ELA achievement four years after enrollment is 
equal to 89 percent of the gap between students who are eligible for free or reduced-price lunch and those 
who are not. 
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Results from alternative approaches for estimating impacts 


As described in the methods chapter, we analyzed the data in nine other ways to ensure that the findings 
were not simply a result of the analytic decisions we made. For example, our primary analysis gave equal 
weighting to students, but we also conducted an analysis that gave equal weighting to schools and another 


that gave equal weighting to cohorts. 


The results of our analyses were similar whether weighting equally by student, cohort, or school, for both 
math and ELA and across all years. All cohorts and schools had strong, positive, and statistically 
significant impacts, and none appeared to drive the strong results for the full sample. Moreover, all the 
analyses resulted in statistically significant impacts in math and ELA for all four years. 


Figure 8 shows the adjusted difference in the average math z-scores between students enrolled in TurnNJ 
schools and the matched students who enrolled in other schools. The red “x” represents the results from 
our primary analysis, and the green dots represent the results from the alternative approaches. Most of the 
alternative approaches resulted in differences between the TurnNJ and comparison students in math 
achievement close to 1 standard deviation in the first two years. The findings for the three-year and four- 
year impacts from the alternative approaches vary more because the sample sizes are smaller, but all 


approaches still found large and statistically significant impacts. 


Figure 8. TurnNJ impacts on math achievement, by alternative approach to the analysis and years 


after enrollment 


14 

1.2 

e 
e 
e % 
1.0 * 
we Xe og @ e 
“wo e ~ x 
bd e e 

= 0.8 fe 
= C4 
i 0 & e 
8 ° 
£ 06 

0.4 

0.2 

oe 1 year after enrollment 2 years after enrollment 3 years after enrollment 4 years after enrollment 
Source: Administrative data from the New Jersey Department of Education from the 2013-2014 to 2018-2019 

school years. 

Note: This figure reports the estimated regression-adjusted differences in mean math achievement between the 
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TurnNJ students and comparison students. The difference in means is reported in standard deviation units. 
The “x” indicates the impact estimate for the primary approach, and the dots indicate impact estimates for 
alternative approaches. A description of each alternative approach is provided in Appendix C. 
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The pattern of results in ELA across the alternative approaches was similar. Most alternative approaches 
resulted in differences between the TurnNJ and comparison students that were close to the estimates from 
our primary analysis (Figure 9). At four years after enrollment, the red “x” representing the estimate from 
our primary analysis is lower than the estimates from seven of the nine approaches, suggesting that our 
benchmark Year 4 estimate may be conservative. Overall, the results from the alternative approaches 
suggest that the large gains that students experienced from enrolling in TurnNJ schools were not inflated 
due to the analytic decisions that we made. 


ey 
Figure 9. TurnNJ impacts on English language arts achievement, by alternative approach to the 
analysis and years after enrollment 
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Administrative data from the New Jersey Department of Education from the 2013-2014 to 2018-2019 
school years. 


This figure reports the estimated regression-adjusted differences in mean English language arts 
achievement between the TurnNJ students and comparison students. The difference in means is reported 
in standard deviation units. The “x” indicates the impact estimate for the primary approach, and the dots 
indicate impact estimates for alternative approaches. A description of each alternative approach is provided 
in Appendix C. 
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Box 3. Fidelity of implementation of Uncommon model in TurnNJ schools 


Goal. The implementation analysis was intended to assess whether the key components of the Uncommon 
school model were implemented with fidelity in the TurnNJ schools. 


Data and methods. To assess implementation fidelity, we used (1) survey data from Uncommon staff for 
the 2018-2019 and 2019-2020 school years and (2) bell times and academic calendars for the 2018-2019 
and 2019-2020 school years. Uncommon administers an annual survey to school staff; we added items to 
the survey instruments to collect data for the study. Survey data included all school instructional staff in 
the TurnNJ schools and all other schools in Uncommon’s Newark and Camden regions. 


Measures for each of the six components were developed from these data as follows: 


1. College preparatory mission: single survey item 

2. High standards for academics and character: composite measure using multiple survey items 
3. Highly structured learning environment: composite measure using multiple survey items 
4 


Longer school day and longer school year: composite measure based on the daily bell times and 
academic calendars for each school 


un 


Focus on accountability and data-driven instruction: single survey item 


6. Faculty of committed and talented leaders and teachers: composite measure using multiple survey 
items 


Separately for each of the six components, we then compared the mean value for each of the four TurnNJ 
schools to the mean value for all non-TurnNJ Uncommon elementary schools and middle schools in New 
Jersey (10 schools in 2018-2019 and 11 schools in 2019-2020). 


Results. Most components of the Uncommon school model, as measured by our data sources, were 
implemented similarly in the TurnNJ schools and in other Uncommon schools in New Jersey. During the 
2018-2019 school year, across four of the five components measured using survey data, a majority of 
TurnNJ schools either equaled or exceeded the non-TurnNJ mean, indicating that the TurnNJ schools 
were successful in achieving the same quality of implementation of standards, staff, mission, and data use 
relative to other Uncommon schools. For the fifth component—highly structured learning environment— 
measured using survey data, two TurnNJ schools were below the Uncommon mean. For instructional 
time, there were no differences between TurnNJ schools and other schools in New Jersey. During the 
2019-2020 school year, a majority of the TurnNJ schools had mean values for all six components that 
were approaching the mean for the non-TurnNJ schools, but they equaled or exceeded the non-TurnNJ 
mean across only two components. The results were similar when all four TurnNJ schools were included 
and when only the three TurnNJ schools in the impact sample were included. 


Implications. The findings suggest the Uncommon school model was largely implemented with fidelity 
in the TurnNJ schools during the period in which we measured outcomes, and the estimated impacts on 
students reflect the impacts of the TurnNJ project as intended. Notably, the TurnNJ schools appeared to 
be implementing the Uncommon school model within three to five years of opening nearly as consistently 
as other Uncommon schools. 


(A detailed discussion of the implementation fidelity analysis is in Appendix D.) 
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Discussion 


This study expanded upon prior research by showing that the positive effects of attending Uncommon 
schools on students’ math and ELA achievement observed in other studies extended to those attending 
three of Uncommon’s TurnNJ schools (Furgeson et al. 2012; Teh et al. 2010; Woodworth and Raymond 
2013, Woodworth et al. 2017). It also expanded on the limited literature on the effectiveness of different 
types of school turnaround efforts. 


The results of this study were on the high end of standardized impacts in math and ELA that have been 
reported in other studies of turnaround initiatives. Charter-managed turnaround efforts in the New Orleans 
Recovery School District, Boston Public Schools, and Loche High School in the LA Unified School 
District led to positive impacts in ELA and math (Abdulkadiroglu et al. 2016; Harris and Larsen 2016; 
Herman et al. 2012), but other charter-managed efforts in Tennessee’s Achievement School District and 
the School District of Philadelphia showed no significant impacts (Zimmer et al. 2017; Hallgren et al. 
2020). Across these studies, the greatest differences between students attending the turnaround schools 
and comparison students were approximately 0.4 standard deviations for one year after enrollment in both 
math and ELA (Harris and Larsen 2016). In the current study, we observed one-year impacts of 1.0 
standard deviation in math and 0.7 standard deviations in ELA. These results provide promising evidence 
of the effectiveness of three TurnNJ schools managed by Uncommon in improving students’ academic 
outcomes. 


At the same time, we should exercise caution when extending the results beyond the schools in this study 
because the study used a small sample size in two similar settings and did not include a comprehensive 
evaluation of the implementation of the TurnNJ model. It is unclear whether the TurnNJ model would be 
as successful in other schools and districts. Although our implementation analysis showed that the 
TumNJ schools implemented most components of the Uncommon school model similarly to other non- 
TumNJ schools in the Uncommon network, the study did not examine how these components contribute 
to strong academic achievement. 
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Conclusions 


Students who enrolled in a TurnNJ school had higher average achievement in math and ELA than similar 
students who attended other schools for all four years after their initial enrollment in a TurnNJ school. 
After only one year from their initial enrollment, TurnNJ students were approximately 1.8 years ahead of 
matched comparison students in both subjects. After four years, TurnNJ students’ learning gains relative 
to comparison students were equivalent to approximately 2.1 years in math and 1.6 years in ELA. These 
results were consistent across the nine alternative approaches to our analysis. 


Since the time of the study, Uncommon TurnNJ has expanded to serve more students in additional 
schools and grade levels. A new elementary school opened in Camden in 2017-2018 for kindergarten and 
lst grade and was built out to grade 4 by 2020-2021. In addition, a new high school in Camden opened in 
the 2020-2021 school year for 9th graders and will build out one additional grade level per year until it 
serves all four grades. 


The expansion of TurnNJ provides opportunities for more research that can further our understanding of 
the impacts of TurnNJ in more schools and grade levels. This research should include learning more 
about how schools are implementing the six core components of the Uncommon school model and which 
are most promising in a turnaround setting. Uncommon could contribute to the field by disseminating 
information on the TurnNJ approach that can be used by districts in their efforts to turn around 
persistently low-performing schools. 


Going forward, Uncommon plans to implement some changes to its approach in schools, including 
modifications to the behavior system, additional trainings for teachers on culturally responsive 
instruction, and curricular changes to better reflect the cultures and identities of their students. Future 
research will be useful for understanding the impacts of changes to Uncommon’s approach and the 
impacts of the Uncommon school model across a broader range of outcomes. It will be important to look 
beyond academic outcomes—particularly standardized tests—to social-emotional outcomes, and to 
examine how students experience instruction in TurnNJ and other Uncommon schools and how they 
perceive themselves as learners. 


Mathematica 


This page has been left blank for double-sided copying. 


Impacts of Uncommon Schools in a Turnaround Setting 


References 


Abdulkadiroglu, Atila, Joshua D. Angrist, Peter D. Hull, and Parag A. Pathak. “Charters Without 
Lotteries: Testing Takeovers in New Orleans and Boston.” American Economic Review, vol. 106, no. 
7, 2016, pp. 1878-1920. 


Bloom, H. S., C. J. Hill, A. R. Black, and M. W. Lipsey. “Performance Trajectories and Performance 
Gaps as Achievement Effect-Size Benchmarks for Educational Interventions.” Journal of Research 
on Educational Effectiveness, vol. 1, no. 4, 2008, pp. 289-328. 


Deke, J. and Finucane, M. Moving Beyond Statistical Significance: the BASIE (BAyeSian Interpretation 
of Estimates) Framework for Interpreting Findings from Impact Evaluations (OPRE Report 2019 35). 
Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and 
Families, U.S. Department of Health and Human Services, 2019. 


Furgeson, J., B. Gill, J. Haimson, A. Killewald, M. McCullough, I. Nichols-Barrer, B. Teh, et al. 
“Charter-School Management Organizations: Diverse Strategies and Diverse Student Impacts.” 
Washington, DC: Mathematica Policy Research and Center on Reinventing Public Education, 2012. 


Greenland, S., Senn, S.J., Rothman, K.J. et al. “Statistical tests, P values, confidence intervals, and power: 
a guide to misinterpretations.” European Journal of Epidemiology, vol. 31, 2016, pp. 337-350. 


Hallgren, Kristin, Paul Burkander, and Lauren Scher. “Evaluation of the School District of Philadelphia’s 
Renaissance Initiative and Turnaround Network.” Presentation to the School District of Philadelphia, 
September 2020. 


Harris, Douglas N., and Matthew Larsen. “The Effects of the New Orleans Post-Katrina School Reforms 
on Student Academic Outcomes.” Technical Report. New Orleans, LA: Education Research Alliance 
for New Orleans, February 2016. 


Herman, Joan L., Jia Wang, Jordan Rickles, Vivian Hsu, Scott Monroe, Seth Leon, and Rolf Straubhaar. 
“Evaluation of Green Dot’s Locke Transformation Project: Findings for Cohort | and 2 Students.” 
Los Angeles, CA: National Center for Research on Evaluation, Standards, and Student Testing 
(CRESST), University of California, Los Angeles, May 2012. 

Redding, Christopher, and Tuan D. Nguyen. “The Relationship Between School Turnaround and Student 
Outcomes: A Meta-Analysis.” Educational Evaluation and Policy Analysis, vol. 42, no. 4, 2020, pp. 
493-519. 

Teh, B., M. McCullough, and B. P. Gill. “Student Achievement in New York City Middle Schools 


Affiliated with Achievement First and Uncommon Schools.” Princeton, NJ: Mathematica Policy 
Research, 2010. 


Wasserstein, Ronald L. and Nicole A. Lazar. “The ASA's Statement on p-Values: Context, Process, and 
Purpose.” The American Statistician, vol. 70, no. 2, 2016, pp. 129-133. 

Woodworth, J. L., and M. E. Raymond. “Charter School Growth and Replication.” Palo Alto, CA: 
Stanford University Center for Research on Education Outcomes (CREDO), 2013. 


Woodworth, J.L., M.E. Raymond, C. Han, Y. Negassi, W. P. Richardson, and W. Snow. “Charter 
Management Organizations 2017.” Palo Alto, CA: Stanford University Center for Research on 
Education Outcomes (CREDO), 2017. 


Zimmer, Ron, Gary T. Henry, and Adam Kho. “The Effects of School Turnaround in Tennessee’s 
Achievement School District and Innovation Zones.” Educational Evaluation and Policy Analysis, 
vol. 39, no. 4, 2017, pp. 670-696. 


Mathematica 21 


This page has been left blank for double-sided copying. 


Appendix A 
Data and Methods 


This page has been left blank for double-sided copying. 


Impacts of Uncommon Schools in a Turnaround Setting 


In this appendix, we provide technical details on the data, sample, and methods used for the primary 
analysis to determine the impacts of enrolling in an Uncommon TurnNJ school. 


Data 


Using data obtained from the New Jersey Department of Education for the 2013-2014 through 2018— 
2019 school years, we created the variables for our analysis shown in Table A.1.’ We used each of these 
variables in the models for both propensity score matching and conducting the impact analyses. 


i ae 
Table A.1 Student-level variables constructed for analysis 


Math z-score Continuous Standardized math test scores were normed to have a mean of 0 and 
SD of 1. Test scores were normed by grade level, school year, and city 
(Newark or Camden). Test scores from 2013-2014 were from the New 
Jersey Assessment of Skills and Knowledge (NJASK), test scores from 
2014-2015 through 2017-2018 were from the Partnership for 
Assessment of Readiness for College and Careers (PARCC), and test 
scores from 2018-19 were from the New Jersey Student Learning 
Assessments (NJSLA). A separate value was calculated for each school 
year for students in grades 3-8. 


Reading z-score Continuous Standardized reading test scores were normed to have a mean of 0 and 
SD of 1. Test scores were normed by grade level, school year, and city 
(Newark or Camden). Test scores from 2013-2014 were from the 
NJASK, test scores from 2014-2015 through 2017-2018 were from the 
PARCC, and test scores from 2018-19 were from the NJSLA. A separate 
value was calculated for each school year for students in grades 3-8. 


Alternative Binary (0/1) An indicator was set to 0 if a student did not take an alternative 
assessment assessment in a given school year and was equal to 1 if a student took 
an alternative assessment in a given school year. An alternative 
assessment could be either the APA, given to special education 
students, or Access for ELLs, given to English learners. A separate value 
was reported for each school year for students in grades 3-8. 


Accommodation Binary (0/1) An indicator was set to 0 if a student did not have a testing 
accommodation in a given school year and was equal to 1 if a student 
was given a testing accommodation in a given school year. A separate 
value was reported for each school year for students in grades 3-8. 


Female? Binary (0/1) An indicator was set to 0 if gender was male and set to 1 if gender was 
female. 
Hispanic Binary (0/1) An indicator was set to 0 if ethnicity was not Hispanic/Latino and set to 1 


if ethnicity was Hispanic/Latino. 


7 The data vendor for the New Jersey Department of Education indicated the need to exercise caution when 
reviewing the non-state assessment score data provided for CPMES in the 2015-2016 school year due to submission 
errors. (The state assessment scores are submitted separately by the testing vendor.) As such, we took several steps 
to check the data and examine the robustness of our findings to the inclusion of these data. First, we reviewed the 
relevant data and conducted diagnostics comparing values for the affected demographic and enrollment variables for 
students in the cohort to values for the same cohort of students in adjacent years. We found no notable discrepancies 
in values across years. We then enlisted a programmer external to the study team to conduct an independent review 
of the data and found similar results. Finally, we estimated impacts excluding the relevant data. We found similar 
impacts when (1) estimating impacts excluding all students in the 2015-2016 CPMES cohort from the sample and 
comparing them to impacts that included these students in the sample and (2) estimating impacts for students in the 
2015-2016 CPMES cohort for the 2015-2016 school year and comparing them to impacts estimated for subsequent 
school years. 
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Black? Binary (0/1) An indicator was set to 0 if race was not Black and set to 1 if race was 
Black. The non-Black races included White, Asian, Pacific Islander, and 
American Indian. 


Ever FRPL Binary (0/1) An indicator was set to 0 if the student was never eligible to receive free 
or reduced-price lunch and was set to 1 if the student was ever eligible to 
receive free or reduced-price lunch in any baseline school year. 


Ever retained Binary (0/1) An indicator was set to 0 if the student was never retained and was set 
to 1 if the student was retained in any baseline school year. 


Ever SPED Binary (0/1) An indicator was set to 0 if the student was never eligible to receive 
special education services and was set to 1 if the student was ever 
eligible to receive special education services in any baseline school year. 


Ever ELL Binary (0/1) An indicator was set to 0 if the student was never classified as an 
English learner (EL) and was set to 1 if the student was classified as ELL 
in any baseline school year. 


Ever chronic absentee | Binary (0/1) An indicator was set to 0 if the student was never classified as a chronic 
absentee and was set to 1 if the student was classified as a chronic 
absentee in any baseline school year. A student was classified as a 
chronic absentee when missing at least 10 percent of school days ina 
given school year. 


Ever Uncommon Binary (0/1) An indicator was set to 0 if the student never attended an Uncommon 
school and was set to 1 if the student attended an Uncommon school in 
any school year. This variable was used to define the treatment and 
comparison groups. 


Entry year Continuous The school year in which a student began attending a Turn NJ school or 
an eligible comparison school. 


Entry grade Continuous The grade level in which a student began attending a Turn NJ school or 
an eligible comparison school. 


4 This variable was considered time invariant. We used data from the baseline year if reported. If not reported, we 
obtained data from another year of available data, starting with one year before the baseline year, then one year 
following the baseline year, then two years before the baseline year, and so on, until a value was identified. 


Sample 


Tables A.2 and A.3 show the sample sizes for math and English language arts from one to four years 
following enrollment. We report three sample sizes for the TurnNJ and comparison students: those who 
were in eligible cohort schools and grade levels, those who had required data and were eligible for 
matching, and those who were matched and included in the impact estimates. 


We excluded all those who did not have data on the demographic characteristics and standardized test 
scores required for the analyses. Our analysis included students who repeated grades in any of the years of 
the study. For example, if a student who started in a 4th-grade cohort repeated Sth grade, we would report 
their grade 4 outcomes in Year 1, grade 5 outcomes in Years 2 and 3, and grade 6 outcomes in Year 4. 
We would not report this student’s grade 7 outcomes. Grade repetition was infrequent, and the rates were 
similar across TurnNJ students and comparison students. For example, among students included in the 
analysis of one-year math impacts, 4 percent of TurnNJ students and 4 percent of comparison students 
were repeating their baseline year grade. Across all outcome samples, grade repetition rates for TurnNJ 
students ranged from 0 to 5 percent, and the difference in grade repetition rates between TurnNJ and 
comparison students ranged from less than 1 percentage point to 4 percentage points. For the two analysis 
samples with retention rate differences of more than 2 percentage points (four-year math and reading 
outcomes), a larger percentage of comparison students were retained relative to TurnNJ students. 
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Table A.2. Number of students for the impact estimates in math, by years after enrollment 


Students in initial cohort | Students with required Students matched and 


Yel aTotod wr: late Me |= Le (=) data and eligible for axed Lere(=Xom lamina) ex: (eas 
levels? raareieqal tare | estimates 
after enrollment TurnNJ /|Comparison| TurnNJ |Comparison| TurnNJ alsa 
1 year 159 10,744 8,417 
2 years 108 7,503 62 6,050 59 452 
3 years 79 5,090 47 4,088 44 328 
4 years 58 3,878 34 3,079 32 235 


Source: Administrative data from the New Jersey Department of Education from the 2013-2014 to 2018-2019 
school years. 


Note: As shown in Figure 5 in the report, the outcome years pool students across cohort years and grade levels. 


4 The number of students in the TurnNJ initial cohorts excludes those who had previously attended an Uncommon 
school. 


————— aay 
Table A.3. Number of students for the impact estimates in English language arts, by years after 


enrollment 
Students in initial cohort | Students with required Students matched and 
FXe1 aLoto) wr: lale me] e-Le(:) data and eligible for Takes ere (=Xom lamina) ex: (eas 
levels? raareieqal tare | estimates 
after enrollment TurnNJ /|Comparison| TurnNJ |Comparison| TurnNJ tile 
1 year 159 10,744 8,417 
2 years 108 7,503 62 6,077 59 451 
3 years 79 5,090 47 4,124 45 318 
4 years 58 3,878 35 3,085 33 256 


Source: Administrative data from the New Jersey Department of Education from the 2013-2014 to 2018-2019 
school years. 


Note: As shown in Figure 5 in the report, the outcome years pool students across cohort years and grade levels. 


4 The number of students in the TurnNJ initial cohorts excludes those who had previously attended an Uncommon 
school. 
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Methods 


In this section, we provide additional details on the propensity score matching procedure used to create a 
balanced sample and the model used to estimate impacts on math and English language arts achievement. 
We implemented these procedures in R version 3.5.1, using the MatchIt and glmnet packages, and Stata 
version 16.1. 


Propensity score model 


We used a logistic model to estimate the probability that a student enrolled in a TurnNJ school, as 
follows: 


logit(T;) = a+ BX; 


where 7 is an indicator for ever enrolling in a TurmnNJ school for student i; X is a vector of student 
covariates including students’ race and ethnicity, gender, special education status, free or reduced-price 
lunch eligibility, English learner status, chronic absentee status, retention status, alternative assessment 
status, testing accommodation status, grade level, school year, and baseline math and ELA test scores; 
and a and f are parameters to be estimated. We ran separate models by school level (elementary or 
middle), city (Camden or Newark), and outcome sample, as there were some students who had the 
required math scores for analysis but not the required ELA scores, or vice versa. However, we used the 
same specifications across all the models to ensure a consistent approach in our analysis. 


We tested variations of the model specification and selected the specification that achieved the greatest 
level of covariate balance across the separate models for each combination of school level and city while 
maintaining a reasonable sample size. The features we varied included quadratic terms for the 
standardized test scores, models with interaction terms, models with fixed effects for the school attended 
in the baseline year, and models with school-level baseline covariates. The school-level covariates 
included an indicator for whether the baseline school was a traditional public school or a charter, and 
school averages of the student-level variables, including the school average z-score in math and English 
language arts and the percentage of students in the school with each of the demographic characteristics. 


We ultimately selected the model that included one interaction term: Black and math z-score. Because we 
did not have theory-based hypotheses about interactions, we tested all possible two-way interactions using 
a procedure called least absolute shrinkage and selection operator (LASSO) regression with the R 
package glmnet. The program identified several interactions across the models, but Black and math z- 
score was the interaction that was identified most frequently across the separate propensity score models. 
We did not include quadratic terms, fixed effects, or school-level covariates. The variations of the models 
with fixed effects and with school-level covariates tended to explain a much greater proportion of the 
variation in enrollment in TurnNJ schools but resulted in very few matches and low levels of balance on 
student-level covariates. 


Matching method 
The matching method for our benchmark approach had the following features: 


e Nearest neighbor matching: Each TurnNJ student was matched to comparison students with the 
nearest propensity scores. 


e 10:1 ratio matching: Each TurnNJ student could be matched with up to 10 comparison students. 
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e Matching with replacement: Once matched, a comparison student could be matched again. 


e Caliper: All matches were required to have a maximum distance, known as a “caliper,” of 0.2 
standard deviations of a propensity score. If a TurnNJ student had a greater distance to its nearest 
comparison student, the student was not matched. 


Before selecting this approach, we tested other variations of matching procedures, including matching 
without replacement, different matching ratios, and using optimal matching, rather than nearest neighbor 
matching. Optimal matching is a different approach to nearest neighbor matching where instead of 
selecting the nearest neighbor to each treatment observation, the algorithm seeks to minimize an overall 
distance measure across all matched pairs. Our selected approach achieved better covariate balance than 
the other approaches. The results from other matching approaches are provided in Appendix C. 


Diagnostics 


To select our propensity model and matching method, we conducted two types of diagnostics: covariate 
balance and overlap in propensity scores. We assessed covariate balance by calculating absolute 
standardized mean differences on all covariates between TurnNJ and matched comparison students. We 
prioritized achieving covariate balance on all baseline standardized test scores and ensured that they all 
had absolute standardized mean differences well below 0.25 to comply with What Works Clearinghouse 
standards. Tables A.4 through A.11 show the means and standardized mean differences for TurnNJ and 
comparison students on math and English language arts test scores in the year before enrollment. Tables 
A.12 through A.19 show the percentage of TurnNJ and comparison group students with each 
demographic characteristic. In both types of tables, we show both the matched comparison group and the 
larger pool of comparison students before matching (shaded in gray). This illustrates the improvement in 
creating balanced samples through matching. 


We reviewed the overlap in propensity scores in the treatment and control groups using jitter plots that 
divided the sample into four groups: unmatched treatment units, matched treatment units, matched control 
units, and unmatched control units (see example in Figure A.1). We ensured that all the matched TurnNJ 
and comparison students (in the middle panels) had overlapping propensity scores. 
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Table A.4. Math and English language arts achievement at baseline for students included in the 
one-year math impact estimates 


Comparison Comparison 
students in students in 
(or-Taale(=1alr-lale| oF-Tantel(-Tamr-lale| 

Newark: After Newark: Before 


Difference between 
tre | students Harel cea tare} Harel cea iare} TurnNJ and matched 


N = 724 N = 8,417 comparison students 
Test subject 7 ae eae Standardized | p-value of 
baseline difference | difference 
Math -0.39 0.93 -0.35 0.92 0.03 0.99 -0.05 0.67 


ELA -0.27 0.90 -0.26 0.90 0.01 0.99 -0.01 0.92 


Source: Administrative data from the New Jersey Department of Education from the 2013-2014 to 2018-2019 
school years. 


Note: Standardized test scores were normalized to have a mean of 0 and standard deviation of 1, by grade level, 
city (Camden or Newark), and school year. 


ELA = English language arts; SD = standard deviation. 


| 
Table A.5. Math and English language arts achievement at baseline for students included in the 
one-year English language arts impact estimates 


Comparison Comparison 
students in students in 
(or-Tinte(-ame-lare fer-Tinte(-amr-lare 


Newark: After Newark: Before Difference between 


Pare students matching arclceq alate} TurnNJ and matched 


N=714 N = 8,417 comparison students 


Test subject _—_ Standardized} p-value of 
baseline difference | difference 


Math -0.39 0.93 -0.34 0.95 0.03 0.99 -0.06 0.61 


ELA -0.27 0.90 -0.23 0.92 0.01 0.99 -0.05 0.69 


Source: Administrative data from the New Jersey Department of Education from the 2013-2014 to 2018-2019 
school years. 


Note: Standardized test scores were normalized to have a mean of 0 and standard deviation of 1, by grade level, 
city (Camden or Newark), and school year. 


ELA = English language arts; SD = standard deviation. 
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Table A.6. Math and English language arts achievement at baseline for students included in the 
two-year math impact estimates 


Comparison Comparison 

students in students in 

(oF-Trivel-Vamr-lale| (or-Tiile(-Tamr- lave, 
TurnNJ Newark: After Newark: Before 


Difference between 
com Sie Ste TurnNJ and matched 


comparison students 
Test subject at —N=S0se {compari p-value of 
baseline difference difference 
Math -0.29 0.94 -0.23 0.96 0.04 0.99 -0.06 0.67 


ELA -0.24 0.92 -0.21 0.97 0.02 0.98 -0.03 0.84 


Source: Administrative data from the New Jersey Department of Education from the 2013-2014 to 2018-2019 
school years. 


Note: Standardized test scores were normalized to have a mean of 0 and standard deviation of 1, by grade level, 
city (Camden or Newark), and school year. 


ELA = English language arts; SD = standard deviation. 


_————————————————— | 
Table A.7. Math and English language arts achievement at baseline for students included in the 
two-year English language arts impact estimates 


Comparison Comparison 
students in students in 
fer Tinte(-Yane-lare| fer-Trite(-Yamr-lare 


TurnNJ Newark: After Newark: Before Difference between 


mom areLeraliale| aareicenal are | TurnNJ and matched 


toy ma comparison students 


Test subject at Standardized| p-value of 
baseline difference difference 


Math -0.29 0.94 -0.28 0.94 0.04 0.99 0.00 0.98 


ELA -0.24 0.92 -0.25 0.94 0.01 0.98 0.01 0.92 


Source: Administrative data from the New Jersey Department of Education from the 2013-2014 to 2018-2019 
school years. 


Note: Standardized test scores were normalized to have a mean of 0 and standard deviation of 1, by grade level, 
city (Camden or Newark), and school year. 


ELA = English language arts; SD = standard deviation. 
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Table A.8. Math and English language arts achievement at baseline for students included in the 
three-year math impact estimates 


Comparison Comparison 

students in students in 

(oF-Trivel-Vamr-lale| (or-Tiile(-Tamr- late, 
TurnNJ Newark: After Newark: Before 


Difference between 
ce tre | ter TurnNJ and matched 


comparison students 
Test subject at Masons _|_sompais p-value of 
baseline difference difference 
Math -0.28 0.96 -0.16 1.04 0.02 0.98 -0.12 0.46 


ELA -0.12 0.88 -0.05 0.96 -0.01 0.96 -0.08 0.61 


Source: Administrative data from the New Jersey Department of Education from the 2013-2014 to 2018-2019 
school years. 


Note: Standardized test scores were normalized to have a mean of 0 and standard deviation of 1, by grade level, 
city (Camden or Newark), and school year. 


ELA = English language arts; SD = standard deviation. 


as 
Table A.9. Math and English language arts achievement at baseline for students included in the 
three-year English language arts impact estimates 


Comparison Comparison 
students in students in 
fer-Tinte(-vane-lare| fer-Taite(-Yamr-lare| 


TurnNJ Newark: After Newark: Before Difference between 


Eten areLeqaliale| aareicenal are | TurnNJ and matched 


N = 318 ae a comparison students 


Test subject at Standardized| p-value of 
baseline difference difference 


Math -0.32 0.98 -0.21 1.07 0.04 0.99 -0.10 0.51 


ELA -0.14 0.88 -0.09 1.04 0.00 0.97 -0.05 0.76 


Source: Administrative data from the New Jersey Department of Education from the 2013-2014 to 2018-2019 
school years. 


Note: Standardized test scores were normalized to have a mean of 0 and standard deviation of 1, by grade level, 
city (Camden or Newark), and school year. 


ELA = English language arts; SD = standard deviation. 
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Table A.10. Math and English language arts achievement at baseline for students included in the 
four-year math impact estimates 


Comparison Comparison 

students in students in 

(oF-Trivel=-VaMr-lale| (or- Tile (-Tamr- lave, 
TurnNJ Newark: After Newark: Before 


Difference between 
ie ire ters TurnNJ and matched 


comparison students 
Test subject at —N=S0re_{_sonparis p-value of 
baseline difference difference 
Math -0.23 0.96 -0.11 1.04 0.00 0.98 -0.12 0.53 


ELA -0.08 0.96 -0.07 1.01 -0.03 0.95 -0.01 0.96 


Source: Administrative data from the New Jersey Department of Education from the 2013-2014 to 2018-2019 
school years. 


Note: Standardized test scores were normalized to have a mean of 0 and standard deviation of 1, by grade level, 
city (Camden or Newark), and school year. 


ELA = English language arts; SD = standard deviation. 


re 
Table A.11. Math and English language arts achievement at baseline for students included in the 
four-year English language arts impact estimates 


Comparison Comparison 
students in students in 
fer-Tinte(-Jane-lare| for-Trate(-Yam-lare| 


TurnNJ Newark: After Newark: Before Difference between 


ioe Ceca ees TurnNJ and matched 


comparison students 
Test subject at —Nasoas _|_sonpais p-value of 
baseline difference difference 
Math -0.27 0.97 -0.25 1.04 0.00 0.98 -0.02 0.93 


ELA -0.10 0.95 -0.08 1.03 -0.03 0.95 -0.02 0.93 


Source: Administrative data from the New Jersey Department of Education from the 2013-2014 to 2018-2019 
school years. 


Note: Standardized test scores were normalized to have a mean of 0 and standard deviation of 1, by grade level, 
city (Camden or Newark), and school year. 


ELA = English language arts; SD = standard deviation. 
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Table A.12. Baseline characteristics for students included in the one-year math impact estimates 


Percentage at baseline 


Comparison Comparison p-value of 
students in students in difference 
(er-Tinte(-ame-lare for-Tinte(-Jae-lare between TurnNJ 
Newark: After Newark: Before and matched 
TurnNJ students matching arclceqallare} comparison 
Characteristic students 
Female 40.0 39.5 50.0 0.93 
Hispanic 18.9 20.2 54.7 0.76 
Black 81.1 79.9 42.4 0.78 
Eligible for free or 
reduced-price lunch 96.7 98.3 93.2 0.30 
Retained 0.0 0.0 3.7 na. 
Special education 13.3 14.4 15.9 0.79 
English learner 3.3 3.9 15.6 0.78 
Chronic absentee 43.3 41.2 19.3 0.70 
Took an alternative 
assessment 2.2 2.1 9.6 0.93 
Received a testing 
accommodation 16.7 19.6 18.7 0.50 


Source: Administrative data from the New Jersey Department of Education from the 2013-2014 to 2018-2019 
school years. 


Note: Chronic absentee is defined as a student missing more than 10 percent of total school days in any baseline 
year. Free or reduced-price lunch eligibility, eligibility for special education services, and English learner 
status were determined based on whether a student ever had such a status in any baseline year of student 
records available. 


n.a. = not applicable. 
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Table A.13. Baseline characteristics for students included in the one-year English language arts 
impact estimates 


Percentage at baseline 


Comparison Comparison p-value of 
students in students in difference 
(oF- Trike (-1aMr-Lale| (or-Trivel=-Vamr-lare| between TurnNJ 
Newark: After Newark: Before and matched 
TurnNJ students matching arclceqaltare} comparison 
Characteristic students 
Female 40.0 40.7 50.0 0.89 
Hispanic 18.9 20.3 54.6 0.75 
Black 81.1 80.8 42.5 0.95 
Eligible for free or 
reduced-price lunch 96.7 98.1 93.2 0.37 
Retained 0.0 0.1 3.7 0.78 
Special education 13.3 13.0 15.9 0.94 
English learner 3.3 4.1 15.5 0.72 
Chronic absentee 43.3 40.9 19.4 0.66 
Took an alternative 
assessment 2.2 3.1 9.6 0.65 
Received a testing 
accommodation 16.7 18.1 18.7 0.74 


Source: Administrative data from the New Jersey Department of Education from the 2013-2014 to 2018-2019 
school years. 


Note: Chronic absentee is defined as a student missing more than 10 percent of total school days in any baseline 
year. Free or reduced-price lunch eligibility, eligibility for special education services, and English learner 
status were determined based on whether a student ever had such a status in any baseline year of student 
records available. 


n.a. = not applicable. 
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Table A.14. Baseline characteristics for students included in the two-year math impact estimates 


Percentage at baseline 


Comparison Comparison p-value of 
students in students in difference 
(oF-Thike(-1amr-Lale| (or-Trivel=Vamr-lale| between TurnNJ 
Newark: After Newark: Before and matched 
TurnNJ students matching areiqaliale| comparison 
Characteristic students 
Female 33.9 33.3 50.1 0.93 
Hispanic 18.6 23.2 53.0 0.43 
Black 81.4 77.9 43.2 0.55 
Eligible for free or 
reduced-price lunch 94.9 95.6 92.4 0.81 
Retained 1.7 2.0 2.7 0.86 
Special education 6.8 6.0 14.9 0.82 
English learner 1.7 2.5 15.1 0.70 
Chronic absentee 33.9 32.4 14.6 0.82 
Took an alternative 
assessment 1.7 2.0 9.8 0.87 
Received a testing 
accommodation 3.4 3.5 14.1 0.96 


Source: Administrative data from the New Jersey Department of Education from the 2013-2014 to 2018-2019 
school years. 

Note: Chronic absentee is defined as a student missing more than 10 percent of total school days in any baseline 
year. Free or reduced-price lunch eligibility, eligibility for special education services, and English learner 
status were determined based on whether a student ever had such a status in any baseline year of student 
records available. 


n.a. = not applicable. 
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Table A.15. Baseline characteristics for students included in the two-year English language arts 
impact estimates 


Percentage at baseline 


Comparison Comparison p-value of 
students in students in difference 
(oF- Trike (1a lr-lare| (or-Trivel=-Vamr-lare| between TurnNJ 
Newark: After Newark: Before and matched 
TurnNJ students matching ari cea altare} comparison 
Characteristic students 
Female 33.9 31.3 50.0 0.69 
Hispanic 18.6 20.6 52.9 0.72 
Black 81.4 78.9 43.3 0.66 
Eligible for free or 
reduced-price lunch 94.9 94.8 92.4 0.96 
Retained 1.7 1.9 Delf 0.93 
Special education 6.8 8.5 14.9 0.65 
English learner 1.7 4.0 15.0 0.39 
Chronic absentee 33.9 31.1 14.6 0.67 
Took an alternative 
assessment 1.7 2.4 9.8 0.74 
Received a testing 
accommodation 3.4 4.9 14.1 0.62 


Source: Administrative data from the New Jersey Department of Education from the 2013-2014 to 2018-2019 
school years. 


Note: Chronic absentee is defined as a student missing more than 10 percent of total school days in any baseline 
year. Free or reduced-price lunch eligibility, eligibility for special education services, and English learner 
status were determined based on whether a student ever had such a status in any baseline year of student 
records available. 


n.a. = not applicable. 
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Table A.16. Baseline characteristics for students included in the three-year math impact estimates 


Percentage at baseline 


Comparison Comparison p-value of 
students in students in difference 
Camden and Camden and between TurnNJ 
Newark: After Newark: Before and matched 
TurnNJ students matching areiraliale| comparison 
Characteristic students 
Female 38.6 39.9 49.7 0.88 
Hispanic 11.4 13.4 49.7 0.71 
Black 88.6 85.7 45.3 0.60 
Eligible for free or 
reduced-price lunch 93.2 93.3 90.9 0.98 
Retained 2.3 3.0 15) 0.78 
Special education 6.8 5.0 12.8 0.61 
English learner 2.3 2.5 13.9 0.93 
Chronic absentee 34.1 30.6 9.2 0.64 
Took an alternative 
assessment 2.3 2.1 11.0 0.94 
Received a testing 
accommodation 2.3 2.0 5.9 0.92 


Source: Administrative data from the New Jersey Department of Education from the 2013-2014 to 2018-2019 
school years. 


Note: Chronic absentee is defined as a student missing more than 10 percent of total school days in any baseline 
year. Free or reduced-price lunch eligibility, eligibility for special education services, and English learner 
status were determined based on whether a student ever had such a status in any baseline year of student 
records available. 


n.a. = not applicable. 
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Table A.17. Baseline characteristics for students included in the three-year English language arts 
impact estimates 


Percentage at baseline 


Comparison Comparison p-value of 
students in students in difference 
Camden and Camden and between TurnNJ 
Newark: After Newark: Before and matched 
TurnNJ students farcical are] matching comparison 
Characteristic students 
Female 37.8 40.7 49.9 0.71 
Hispanic 11.1 12.5 49.9 0.79 
Black 88.9 87.7 45.2 0.82 
Eligible for free or 
reduced-price lunch 93.3 94.2 90.9 0.83 
Retained 2.2 3.4 15) 0.69 
Special education 6.7 4.9 12.7 0.63 
English learner 2.2 1.8 13.8 0.83 
Chronic absentee 35.6 29.3 9.2 0.39 
Took an alternative 
assessment 2.2 1.6 10.9 0.76 
Received a testing 
accommodation 2.2 1.8 5.9 0.85 


Source: Administrative data from the New Jersey Department of Education from the 2013-2014 to 2018-2019 
school years. 


Note: Chronic absentee is defined as a student missing more than 10 percent of total school days in any baseline 
year. Free or reduced-price lunch eligibility, eligibility for special education services, and English learner 
status were determined based on whether a student ever had such a status in any baseline year of student 
records available. 


n.a. = not applicable. 
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Table A.18. Baseline characteristics for students included in the four-year math impact estimates 


Percentage at baseline 


Comparison 

students in 

(or-Trivel=Vamr-lale| 
Newark: Before 


Comparison 
students in 


p-value of 
difference 
between TurnNJ 
F-TaveMait-iceqal-ve| 


(or-Tintel-Janr-lale 
Newark: After 


TurnNJ students matching areLeqaliale| comparison 
Characteristic students 
Female 40.6 42.5 49.8 0.84 
Hispanic 9.4 9.7 47.7 0.96 
Black 90.6 90.9 46.4 0.97 
Eligible for free or 
reduced-price lunch 90.6 92.0 90.0 0.79 
Retained 0.0 0.0 1.1 n.a 
Special education 0.0 0.0 11.4 n.a 
English learner 3.1 2.4 13.4 0.79 
Chronic absentee 34.4 31.7 6.1 0.76 
Took an alternative 
assessment 3.1 1.5 12.2 0.52 
Received a testing 
accommodation 0.0 0.0 1.1 n.a. 


Source: Administrative data from the New Jersey Department of Education from the 2013-2014 to 2018-2019 


school years. 


Note: Chronic absentee is defined as a student missing more than 10 percent of total school days in any baseline 
year. Free or reduced-price lunch eligibility, eligibility for special education services, and English learner 
status were determined based on whether a student ever had such a status in any baseline year of student 


records available. 
n.a. = not applicable 
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Table A.19. Baseline characteristics for students included in the four-year English language arts 
impact estimates 


Percentage at baseline 


Comparison Comparison p-value of 
students in students in difference 
Camden and Camden and between TurnNJ 
Newark: After Newark: Before and matched 
TurnNJ students faareicenal are | matching comparison 
Characteristic students 
Female 39.4 41.0 49.6 0.86 
Hispanic 9.1 9.8 47.6 0.90 
Black 90.9 90.2 46.6 0.90 
Eligible for free or 
reduced-price lunch 90.9 92.6 90.0 0.73 
Retained 0.0 0.0 ql n.a. 
Special education 3.0 4.1 11.4 0.76 
English learner 3.0 4.7 13.4 0.67 
Chronic absentee 33.3 28.4 6.2 0.56 
Took an alternative 
assessment 3.0 3.9 12.2 0.81 
Received a testing 
accommodation 0.0 0.0 1.1 n.a. 


Source: Administrative data from the New Jersey Department of Education from the 2013-2014 to 2018-2019 
school years. 


Note: Chronic absentee is defined as a student missing more than 10 percent of total school days in any baseline 
year. Free or reduced-price lunch eligibility, eligibility for special education services, and English learner 
status were determined based on whether a student ever had such a status in any baseline year of student 
records available. 


n.a. = not applicable. 
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Figure A.1. Distribution of propensity scores for the Camden grade 4 propensity score model 


Unmatched Treatment Units 


Matched Treatment Units 


0.0 0.1 0.2 0.3 0.4 
Propensity Score 
Note: The dots represent the weight a student has in the analysis. All treatment units (TurnNJ students) have a 


weight of 1, so they are all the same size. For the control units, larger dots represent students who were 
matched with multiple TurnNJ students due to matching with replacement. Smaller dots indicate that 
multiple comparison students were matched to the same TurnNJ student. 
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Impact estimates 


We estimated the impacts of TumNJ enrollment using a statistical model that compares the regression- 
adjusted mean math and English language arts outcomes of the TurnNJ and comparison groups. The 
model is as follows: 


yi, =at+ PX, + OT; +e; 


where y represents the outcome (standardized math or ELA score) for student i; X is a vector of student 
covariates including students’ race and ethnicity, gender, special education status, free or reduced-price 
lunch eligibility, English learner status, chronic absentee status, retention status, grade level, school year, 
and baseline math and ELA test scores; 7 is an indicator for ever enrolling in a TurnNJ school; e is a 
student-level error term; and a, f, and 0 are parameters to be estimated, with robust standard errors 
clustered at the school level. In this framework, the @ term represents the impact of enrollment in a 
TurnNJ school. Also, in this framework, larger schools and cohorts have a greater influence on the impact 
estimate than smaller schools and cohorts. We estimated separate models for each combination of subject 
(math or ELA) and duration (1 to 4 years after enrollment). The impact model did not include any 
quadratic or interaction terms. 


We used the analysis weights generated by the MatchIt package in R to account for the features of the 
matching design so that the comparison group is weighted to look like the TurnNJ group. Each TurnNJ 
student and the matched comparison students can be thought of as a group. Within each group, the 
TurnNJ student is given a weight of 1 and the comparison students are given a preliminary weight of 
1/n,, where n, is the number of comparison students in the group. Each comparison student’s weight is 
then added up across the groups in which it was matched. Finally, the comparison weights are rescaled to 
sum to the number of uniquely matched comparison students. These weights enable us to estimate the 
average treatment effect on the treated. 


Conversion to years of learning 


After estimating the impacts of TurnNJ enrollment, we converted the effect sizes to years of learning 
using the following benchmarks derived in Bloom et al. (2008) for average annual student achievement 
gains: 


e From the end of 3rd grade to the end of 4th grade and from the end of 4th grade to the end of 5th 
grade, averaged (for one-year impacts)—0.54 standard deviations (SDs) in math and 0.38 SDs in 
ELA 


e From the end of 3rd grade to the end of 5th grade and from the end of 4th grade to the end of 6th 
grade, averaged (for two-year impacts)—0.55 SDs in math and 0.39 SDs in ELA 


e From the end of 3rd grade to the end of 6th grade and from the end of 4th grade to the end of 7th 
grade, averaged (for three-year impacts)—0.49 SDs in math and 0.36 SDs in ELA 


e From the end of 3rd grade to the end of 7th grade (for four-year impacts) —0.45 SDs in math and 0.33 
SDs in ELA 
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In this appendix, we provide additional details about the results that correspond to Figures 6 and 7 in the 
results chapter of the report. These results include the statistical significance of the impact estimates using 
p-values, with the traditional threshold of p < 0.05. We also discuss an alternate interpretation of the 
impact estimates using Bayesian posterior probabilities and present the likelihood that our impact 
estimates reflect a large positive impact of TurnNJ enrollment on student achievement. 


Detailed results 


Tables B.1 and B.2 report the sample sizes, coefficients, standard errors, and p-values for the math and 
English language arts impacts. To improve the interpretability of the coefficients, we also provide the 
conversion of the coefficients to cumulative years of learning based on benchmarks by Bloom et al. 
(2008) on average learning gains. For example, an impact of 0.98 translates to approximately 1.82 years 
of math learning for students in 4th, 5th, and 6th grade. We also summarize the results across alternative 
analyses by showing the percentage that had statistically significant results and the percentage that had 
impact magnitudes greater than one year of learning. All alternative analyses met these criteria. 


_—_—__—————— ere 
Table B.1. TurnNJ impacts on math achievement, by years after enrollment 


Effect size | Percentage of | Percentage of 


converted to| alternative alternative 
Number of cumulative | analyses with | analyses with 
years after ie laler-lae| years of effect size > 1 Tre laliicer-lai' 
enrollment Coefficient | error EFT al tare | school year | (p < 0.05) result 
1 year 814 0.98** 0.08 <0.001 1.82 100 100 
2 years 511 0.98** 0.09 <0.001 1.77 100 100 
3 years 372 0.72** 0.14 <0.001 1.47 100 100 
4 years 267 0.95** 0.17 <0.001 2.12 100 100 
Note: This table reports the coefficients from linear regressions of standardized math test scores on an indicator 


variable for enrollment in a TurnNJ school. Separate models were run for each outcome year. Standardized 
test scores were normalized to have a mean of 0 and standard deviation of 1, by grade level, city (Camden 
or Newark), and school year. Regression controls include one year of baseline test scores in math and 
English language arts, as well as indicator variables for baseline demographic characteristics. The model 
used analysis weights to reflect the propensity score matching approach. Robust standard errors are 
reported. The effect sizes were converted to years of learning using benchmarks by Bloom et al. (2008) on 
average learning gains. The nine alternative approaches are described in Table C.1 of Appendix C. 


* Significantly different from zero at the .05 level, two-tailed test. 
** Significantly different from zero at the .01 level, two-tailed test. 
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Table B.2. TurnNJ impacts on English language arts achievement, by years after enrollment 


Percentage 
Effect size |of alternative| Percentage of 

converted to; analyses alternative 
Number of cumulative | with effect analyses with 
years after Sie laler-lae| years of size > 1 Tefaliicer-lais 
enrollment Coefficient; error | p-value learning school year | (p < 0.05) result 
1 year 804 0.70** 0.04 <0.001 1.84 100 100 
2 years 510 0.82** 0.05 <0.001 2.11 100 100 
3 years 363 0.72** 0.10 <0.001 1.99 100 100 
4 years 289 0.52** 0.07 <0.001 1.59 100 100 
Note: This table reports the coefficients from linear regressions of standardized math test scores on an indicator 


variable for enrollment in a TurnNJ school. Separate models were run for each outcome year. Standardized 
test scores were normalized to have a mean of 0 and standard deviation of 1, by grade level, city (Camden 
or Newark), and school year. Regression controls include one year of baseline test scores in math and 
English language arts, as well as indicator variables for baseline demographic characteristics. The model 
used analysis weights to reflect the propensity score matching approach. Robust standard errors are 
reported. The effect sizes were converted to years of learning using benchmarks by Bloom et al. (2008) on 
average learning gains. The nine alternative approaches are described in Table C.1 of Appendix C. 


* Significantly different from zero at the .05 level, two-tailed test. 
** Significantly different from zero at the .01 level, two-tailed test. 


Alternate approach to interpreting impact estimates 


We recognize that the estimated impacts are not necessarily the true impacts of enrolling in TurnNJ 
schools; there are inevitable biases in our estimate due to the small sample size and our imperfect ability 
to match TurnNJ to comparison students. We also acknowledge that p-values are frequently 
misinterpreted (Wasserstein and Lazar 2016; Greenland et al. 2016). To provide additional information on 
the level of confidence with which we can posit that TurnNJ positively impacted enrolled students, we 
used an alternate approach to interpreting impact estimates called BASIE (BAyeSian Interpretation of 
Estimates). 


The BASIE approach uses Bayesian methods to directly estimate the probability that the true effect of an 
intervention is of a certain size.* To implement the BASIE approach, we used (1) the impact estimate and 
standard error for the intervention that was evaluated and (2) how common it is for generally similar 
interventions to have effects. The commonality of similar intervention achieving positive effects of 
different sizes is called the prior evidence. For example, to estimate the probability that TurnNJ had a true 
effect of greater than 0.5 standard deviations, we considered both the impact estimate and standard error 
from this study as well as the distribution of effects from studies of other educational interventions, and 
specifically, the frequency of effects greater than 0.50 standard deviations. Under the BASIE approach, 
effect estimates from a particular study that are similar to the prior evidence are judged to be more 
credible; effect estimates that are very different are deemed less credible. 


Using the BASIE approach, in Table B.3 we report the probability that the true impacts of TurnNJ were 
positive at all, the probability that the true impacts were greater than 0.5 standard deviations, and the 
probability that the true impacts were greater than | standard deviation. These probabilities are reported 


8 See Deke and Finucane 2019 for more information on the BASIE approach. 
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for each of our outcome periods. We show that we are more than 99 percent confident that the true 
impacts were positive for both math and ELA and for all years, and we are also more than 99 percent 
confident that the true impacts were greater than 0.5 standard deviations for both subjects in the first two 
years. This, along with the consistent results from the 10 alternative analyses, provides greater confidence 
about the large, positive impacts we observed for TurnNJ. 


a 
Table B.3. Probability that TurnNJ enrollment had a positive impact on students 


Probability that true impact was: 
Estimated Greater than 0 Greater than 0.5 Greater than 1 


Outcome impact SDs SDs iS B) 
One year after enrollment 

ELA achievement 0.70 >0.99 >0.99 0.00 
Math achievement 0.98 >0.99 >0.99 0.05 
ELA achievement 0.82 >0.99 >0.99 0.00 
Math achievement 0.98 >0.99 >0.99 0.04 
ELA achievement 0.72 >0.99 0.89 0.00 
Math achievement 0.72 >0.99 0.62 0.00 
ELA achievement 0.52 >0.99 0.38 0.00 
Math achievement 0.95 >0.99 0.83 0.00 
Note: The probability that the impact was above the specified levels is calculated using the estimated impact, 


estimated standard error, and prior evidence from the What Works Clearinghouse that met standards. 
ELA = English language arts; SD = standard deviation. 
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In this appendix, we describe the alternative approaches we used to estimate the impacts of enrolling in a 


TurnNJ school. 


Alternative approaches that use the same definition of TurnNJ enrollment 


This section provides results for alternative approaches to the same research question as our primary 
approach. These analyses examine the impacts of attending a TurnNJ school on math and ELA 
achievement for students enrolling in a TurnNJ school for the first time. Table C.1 describes the purpose 
of each analysis and explains how it is different from the primary approach. Tables C.2 through C.9 
provide the impact estimates from the primary approach and each alternative approach for every 
subject/year combination. 


SSeS Eee 
Table C.1. Description of alternative analyses 


Name of 
approach 


Feeder school 


PAV ar-lWad(eme(-tel J (ey a) 


that it addresses 
Sample: most 


How the approach is different from the primary approach? 


Restricts the pool of comparison students eligible for matching to only 


restriction restrictive those who attended a baseline school that was also attended by at least 
one TurnNJ student in the baseline year 

Regression Sample: least Estimates impacts using the benchmark impact model without matching in 

model without restrictive advance and limiting the comparison group to matched students 

matching (comparison group includes all eligible students who did not attend a 


Propensity score 
matching without 
a caliper 


Matching approach: 
less restrictive 


TurnNJ or other Uncommon school) 

Removes the caliper so that all TurnNJ students are matched with 
comparison students with the nearest propensity scores, even if their 
propensity scores are more than 0.2 standard deviations apart 


1:1 propensity 
score matching 


Matching approach: 
more restrictive 


TurnNJ students are matched with just 1 comparison student each instead 
of 10 


Optimal matching 


Matching approach: 
different algorithm 


Uses "optimal matching," a matching algorithm that seeks to minimize 
global distance rather than the distance of individual pairs 


Propensity score 


Propensity score 


Uses propensity score weighting rather than matching; all eligible TurnNJ 


weighting method: matching and comparison students are included in the sample; TurnNJ students are 
vs. weighting given a weight equal to the inverse of the propensity score 
Parsimonious Impact model: fewer Limits the independent variables in the impact model to treatment status, 
impact model covariates cohort year, site, and grade level, and the math or English language arts 
z-score from the prior school year (math for math outcomes and English 
language arts for English language arts outcomes) 
Impact model Rescales the analysis weights so that the weights within each TurnNJ 
Impact model weighting: equally school sum to 1 and the weights of the students matched to students 
with equally weighted schools within each TurnNJ school also sum to 1 (the weights of students from the 


weighted schools 


TurnNJ schools sum to 3 and the weights of students from comparison 
schools sum to 3) 


Impact model 
with equally 
weighted cohorts 


Impact model 
weighting: equally 
weighted cohorts 


Rescales the analysis weights so that the weights within each TurnNJ 
cohort sum to 1 and the weights of the students matched to students 
within each TurnNJ cohort also sum to 1 (the weights of students from the 
TurnNJ cohorts sum to 5 and the weights of students from comparison 
schools sum to 5) 


4 The description column explains the aspect of the sensitivity test that is different from the primary approach. All 
other aspects of the alternative approach are the same as the primary, as described in the methods section of the 
report and Appendix A. 
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Table C.2. TurnNJ impacts on math achievement one ‘a after enrollment, by — test 


Benchmark 0.98 0.08 <0.001 
Feeder school restriction 666 1.01 0.08 <0.001 
Regression model without 

matching 8,508 0.95 0.11 <0.001 
Propensity score matching 

without a caliper 856 0.93 0.09 <0.001 
1:1 propensity score matching 173 0.98 0.06 <0.001 
Optimal matching 1,001 0.98 0.08 <0.001 
Propensity score weighting 8,508 0.94 0.10 <0.001 
Parsimonious impact model 814 0.98 0.08 <0.001 
Impact model with equal 

weighting of schools 814 1.06 0.06 <0.001 
Impact model with equal 

weighting of cohorts 814 0.96 0.07 <0.001 


Source: Administrative data from the New Jersey Department of Education from the 2013-2014 to 2018-2019 
school years. 

Note: This table reports the coefficients from linear regressions of standardized math test scores on an indicator 
variable for enrollment in a TurnNJ school. Separate models were run for each outcome year. Standardized 
test scores were normalized to have a mean of 0 and standard deviation of 1, by grade level, city (Camden 
or Newark), and school year. 


Ey 
Table C.3. TurnNJ impacts on English language arts achievement one year after enrollment, by 
picid) test 


Benchmark 0.70 0.04 <0.001 
Feeder school restriction 650 0.72 0.03 <0.001 
Regression model without matching 8,508 0.73 0.05 <0.001 
Propensity score matching without a 

caliper 849 0.70 0.04 <0.001 
1:1 propensity score matching 177 0.87 0.08 <0.001 
Optimal matching 1,001 0.73 0.04 <0.001 
Propensity score weighting 8,508 0.70 0.04 <0.001 
Parsimonious impact model 804 0.70 0.05 <0.001 
Impact model with equal weighting of 

schools 804 0.73 0.04 <0.001 
Impact model with equal weighting of 

cohorts 804 0.75 0.03 <0.001 


Source: Administrative data from the New Jersey Department of Education from the 2013-2014 to 2018-2019 
school years. 

Note: This table reports the coefficients from linear regressions of standardized math test scores on an indicator 
variable for enrollment in a TurnNJ school. Separate models were run for each outcome year. Standardized 
test scores were normalized to have a mean of 0 and standard deviation of 1, by grade level, city (Camden 
or Newark), and school year. 
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Table C.4. TurnNJ impacts on math achievement two years after enrollment, by sensitivity test 


Benchmark 511 0.98 0.09 <0.001 
Feeder school restriction 392 0.98 0.11 <0.001 
Regression model without matching 6,112 0.98 0.12 <0.001 
Propensity score matching without a 

caliper 547 0.97 0.12 <0.001 
1:1 propensity score matching 117 0.98 0.11 <0.001 
Optimal matching 682 0.94 0.11 <0.001 
Propensity score weighting 6,112 0.95 0.10 <0.001 
Parsimonious impact model 511 0.97 0.10 <0.001 
Impact model with equal weighting of 

schools 511 0.95 0.06 <0.001 
Impact model with equal weighting of 

cohorts 511 0.95 0.05 <0.001 


Source: Administrative data from the New Jersey Department of Education from the 2013-2014 to 2018-2019 
school years. 

Note: This table reports the coefficients from linear regressions of standardized math test scores on an indicator 
variable for enrollment in a TurnNJ school. Separate models were run for each outcome year. Standardized 
test scores were normalized to have a mean of 0 and standard deviation of 1, by grade level, city (Camden 
or Newark), and school year. 


_————————_—_—_——— aay 
Table C.5. TurnNJ impacts on English language arts achievement two years after enrollment, by 
sensitivity test 


Benchmark 510 0.82 0.05 <0.001 
Feeder school restriction 385 0.80 0.08 <0.001 
Regression model without 

matching 6,139 0.84 0.08 <0.001 
Propensity score matching 

without a caliper 550 0.82 0.05 <0.001 
1:1 propensity score matching 117 0.91 0.11 <0.001 
Optimal matching 682 0.83 0.06 <0.001 
Propensity score weighting 6,139 0.81 0.06 <0.001 
Parsimonious impact model 510 0.83 0.06 <0.001 
Impact model with equal 

weighting of schools 510 0.81 0.04 <0.001 
Impact model with equal 

weighting of cohorts 510 0.82 0.04 <0.001 


Source: Administrative data from the New Jersey Department of Education from the 2013-2014 to 2018-2019 
school years. 

Note: This table reports the coefficients from linear regressions of standardized math test scores on an indicator 
variable for enrollment in a TurnNJ school. Separate models were run for each outcome year. Standardized 
test scores were normalized to have a mean of 0 and standard deviation of 1, by grade level, city (Camden 
or Newark), and school year. 
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Table C.6. TurnNJ impacts on math achievement three = after enrollment, by sensitivity test 


Benchmark 0.72 0.14 <0.001 
Feeder school restriction 284 0.89 0.14 <0.001 
Regression model without 

matching 4,135 0.81 0.18 <0.001 
Propensity score matching 

without a caliper 403 0.83 0.15 <0.001 
1:1 propensity score matching 87 0.74 0.09 <0.001 
Optimal matching 517 0.76 0.16 <0.001 
Propensity score weighting 4,135 0.75 0.15 <0.001 
Parsimonious impact model 372 0.72 0.14 <0.001 
Impact model with equal 

weighting of schools 372 0.63 0.09 <0.001 
Impact model with equal 

weighting of cohorts 372 0.63 0.09 <0.001 


Source: Administrative data from the New Jersey Department of Education from the 2013-2014 to 2018-2019 
school years. 

Note: This table reports the coefficients from linear regressions of standardized math test scores on an indicator 
variable for enrollment in a TurnNJ school. Separate models were run for each outcome year. Standardized 
test scores were normalized to have a mean of 0 and standard deviation of 1, by grade level, city (Camden 
or Newark), and school year. 


—— es 
Table C.7. TurnNJ impacts on English language arts achievement three years after enrollment, by 
sensitivity test 


N Coefficient Standard error p-value 


Benchmark 363 0.72 0.10 <0.001 
Feeder school restriction 272 0.81 0.11 <0.001 
Regression model without 

matching 4,171 0.81 0.14 <0.001 
Propensity score matching 

without a caliper 394 0.76 0.11 <0.001 
1:1 propensity score matching 86 0.85 0.12 <0.001 
Optimal matching 517 0.76 0.16 <0.001 
Propensity score weighting 4,171 0.80 0.14 <0.001 
Parsimonious impact model 363 0.71 0.10 <0.001 
Impact model with equal 

weighting of schools 363 0.68 0.08 <0.001 
Impact model with equal 

weighting of cohorts 363 0.68 0.08 <0.001 


Source: Administrative data from the New Jersey Department of Education from the 2013-2014 to 2018-2019 
school years. 

Note: This table reports the coefficients from linear regressions of standardized math test scores on an indicator 
variable for enrollment in a TurnNJ school. Separate models were run for each outcome year. Standardized 
test scores were normalized to have a mean of 0 and standard deviation of 1, by grade level, city (Camden 
or Newark), and school year. 
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Table C.8. TurnNJ impacts on math achievement four years after enrollment, by sensitivity test 


N Coefficient | Standard error p-value 
Benchmark 267 0.95 0.17 <0.001 
Feeder school restriction 177 1.15 0.22 <0.001 
Regression model without matching 3,113 1.00 0.17 <0.001 
Propensity score matching without a 
caliper 286 1.04 0.22 <0.001 
1:1 propensity score matching 63 0.70 0.12 <0.001 
Optimal matching 374 0.97 0.19 <0.001 
Propensity score weighting 3,113 1.01 0.18 <0.001 
Parsimonious impact model 267 0.98 0.18 <0.001 
Impact model with equal weighting of 
schools 267 0.87 0.11 <0.001 
Impact model with equal weighting of 
cohorts 267 0.87 0.11 <0.001 


Source: Administrative data from the New Jersey Department of Education from the 2013-2014 to 2018-2019 


school years. 


Note: This table reports the coefficients from linear regressions of standardized math test scores on an indicator 
variable for enrollment in a TurnNJ school. Separate models were run for each outcome year. Standardized 
test scores were normalized to have a mean of 0 and standard deviation of 1, by grade level, city (Camden 
or Newark), and school year. 


Table C.9. TurnNJ impacts on English language arts achievement four years after enrollment, by 
sensitivity test 


N Coefficient | Standard error p-value 
Benchmark 289 0.52 0.07 <0.001 
Feeder school restriction 190 0.65 0.07 <0.001 
Regression model without matching 3,120 0.66 0.04 <0.001 
Propensity score matching without a 
caliper 315 0.71 0.07 <0.001 
1:1 propensity score matching 66 0.75 0.16 <0.001 
Optimal matching 385 0.58 0.07 <0.001 
Propensity score weighting 3,120 0.64 0.05 <0.001 
Parsimonious impact model 289 0.54 0.07 <0.001 
Impact model with equal weighting of 
schools 289 0.51 0.07 <0.001 
Impact model with equal weighting of 
cohorts 289 0.51 0.07 <0.001 


Source: Administrative data from the New Jersey Department of Education from the 2013-2014 to 2018-2019 
school years. 


Note: This table reports the coefficients from linear regressions of standardized math test scores on an indicator 
variable for enrollment in a TurnNJ school. Separate models were run for each outcome year. Standardized 
test scores were normalized to have a mean of 0 and standard deviation of 1, by grade level, city (Camden 
or Newark), and school year. 
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Alternative approach that uses a different definition of TurnNJ enrollment 


To better understand the impacts of TurnNJ enrollment on a more inclusive group of TurnNJ students, we 
also conducted an analysis that included (1) students who enrolled in the TurnNJ elementary schools in 
the first years they opened, (2) students who enrolled in a TurnNJ school for the first time in middle 
school, and (3) students who enrolled in the TurnNJ middle school and had previously attended a TurnNJ 
elementary school during the 2015-2016 school year or later. The goal of this analysis was to examine 
TurnNJ’s impacts for a sample that was as inclusive as possible of all students enrolling in a TumNJ 
school. 


The additional students who attended a TumnNJ elementary school and subsequently enrolled in 5th grade 
at CPMMS in the 2016-2017, 2017-2018, or 2018-2019 school years did not have baseline math and 
ELA test scores that were measured before enrollment in any TurnNJ schools. These students were 
matched to comparison students in Camden who had similar baseline achievement and demographic 
characteristics but did not enroll in CPMMS. 


Tables C.10 and C.11 present findings from this alternative approach. The impacts of TurnNJ enrollment 
on this larger sample of students were positive and significant for both math and ELA one, two, three, and 
four years after enrollment. The magnitudes of the impacts were smaller relative to our primary analysis, 
which included only students enrolling in TurnNJ schools who had baseline math and ELA test scores 
that were measured before enrollment in any TurnNJ or other Uncommon school. This alternative 
analysis likely underestimates the cumulative impact of TurnNJ enrollment because it does not account 
for the impacts of attending a TurnNJ elementary school before enrolling in a TurnNJ middle school. 


ane 
Table C.10. TurnNJ impacts on math achievement for new students and students with prior 
exposure to Uncommon, by years after enrollment 


Number of years 
after enrollment N Coefficient Standard error p-value 


1 year 1,678 0.46 0.13 <0.001 
2 years 983 0.50 0.19 0.012 
3 years 535 0.57 0.16 <0.001 
4 years 272 0.90 0.15 <0.001 
Note: This table reports the coefficients from linear regressions of standardized math test scores on an indicator 


variable for enrollment in a TurnNJ school. Separate models were run for each outcome year. Standardized 
test scores were normalized to have a mean of 0 and standard deviation of 1, by grade level, city (Camden 
or Newark), and school year. 
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Table C.11. TurnNJ impacts on English language arts achievement for new students and students 


with prior exposure to Uncommon, by years after enrollment 


Number of years 
after enrollment N 


Coefficient Standard error p-value 

1 year 1,687 0.31 0.08 <0.001 

2 years 979 0.47 0.12 <0.001 

3 years 533 0.49 0.20 0.015 

4 years 294 0.56 0.06 <0.001 

Note: This table reports the coefficients from linear regressions of standardized math test scores on an indicator 
variable for enrollment in a TurnNJ school. Separate models were run for each outcome year. Standardized 
test scores were normalized to have a mean of 0 and standard deviation of 1, by grade level, city (Camden 
or Newark), and school year. 
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Impacts of Uncommon Schools in a Turnaround Setting 


In keeping with the requirements of the i3 grant and to provide context for interpreting the estimates of 
the TurnNJ model’s effectiveness, we measured the fidelity with which the key components of the 
Uncommon school model were implemented in the four TurnNJ schools supported by the grant. 
Specifically, we examined the fidelity of implementation of the six components identified in the logic 
model: (1) college preparatory mission, (2) high standards for academics and character, (3) highly 
structured learning environment, (4) longer school day and longer school year, (5) focus on accountability 
and data-driven instruction, and (6) faculty of committed and talented leaders and teachers (Figure 1). 


Component measures 


To measure the implementation of each component, we used two main data sources (Table D.1). First, we 
collected data for the 2018-2019 and 2019-2020 school years from an annual staff survey administered to 
staff in Uncommon schools. Our analysis included responses from school instructional staff in the four 
TumNJ schools and all other elementary and middle schools in Uncommon’s New Jersey regions (455 
educators in 14 schools during the 2018-2019 school year and 477 educators in 15 schools during the 
2019-2020 school year), aggregated to the school level. We used single survey items to measure college 
preparatory mission and focus on accountability and data-driven instruction. We constructed composite 
measures using multiple survey items for three components: high standards for academics and character, 
highly structured learning environment, and faculty of committed and talented leaders and teachers. 
Composite measures were constructed as simple averages of the values for each survey item. We used 
principal components analysis to confirm that each composite was unidimensional and computed the 
standardized Cronbach’s alpha to confirm that each measure was internally consistent, or reliable. 


We also collected bell times and academic calendars for the four TurnNJ schools and all other elementary 
and middle schools in Uncommon’s New Jersey regions for the 2018-2019 and 2019-2020 school years. 
We used these data to construct a measure of annual instructional time for each school. 


All component measures relied on extant data and were not intended to be comprehensive measures of 
each component of the Uncommon school model. Rather, the measures attempted to capture 
characteristics of the school environment that align conceptually with each component. 
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Table D.1. Measures of Uncommon school model components 


UT atexeyealsnredam-xeqicete)| Cronbach’s 
raete(-1mexe)aa) eXeyal-val! Measure alpha? 


Single school staff survey item: 
College preparatory — 
nissan 1. School staff share a mission to prepare all students to succeed na. 
in college. 
Composite of school staff survey items: 
1. My principal keeps the school focused on student achievement 
and conveys a sense of urgency. 
pee | ee 
academics and : 0.94 
character 3. My primary instructional leader meets with me at least 2-3 
times per month to give me feedback on my instruction, plan for 
the upcoming week, or do a general check-in. 
4. My primary instructional leader gives me quality support and 
feedback that improves my ability to raise student achievement. 
Composite of school staff survey items: 
1. Operational systems at my school run smoothly and maximize 
the time students spend learning. 
Highly structured 2. | believe that the school's code of conduct and discipline 0.81 
learning environment systems make every attempt to minimize time spent by , 
teachers on disciplinary matters. 
3. The school's dean of students is effective in managing the 
school culture and discipline systems. 
Longer school day and | Instructional hours per year, constructed using school bell times and “— 
longer school year academic calendars _ 
Focus on Single school staff survey item: 
accountability and 1. My primary instructional leader helps me to use data to drive n.a. 
data-driven instruction instruction and raise student achievement. 
Composite of school staff survey items: 
1. This school year, | have had opportunities at work to learn and 
grow. 
2. There is at least one person on the school's leadership team 
: who encourages my development. 
Faculty of committed 3. |am comfortable going to one of my school leaders to raise 
and talented teachers 0.95 
and leaders aati : ; 
4. | know what is expected of me in order to be successful at work. 
5. Staff morale at the school is positive. 
6. Inthe last month, my principal and/or director of operations 
recognized or praised my work. 
7. Overall, | am satisfied at my school. 


Note: All survey items asked the respondent to provide a rating on a Likert scale of agreement between 1 and 5, 
where 1 indicated strong disagreement with the statement, 2 indicated disagreement with the statement, 3 
indicated neither agreement nor disagreement with the statement, 4 indicated agreement with the 
statement, and 5 indicated strong agreement with the statement. 


4 For all three composite measures, the alphas exceeded the conventional standard for reliability of greater than 0.7. 
n.a. = not applicable. 
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Fidelity assessment 


To assess whether the TurnNJ schools implemented each component with fidelity, we used non-TurnNJ 
Uncommon elementary and middle schools in New Jersey (10 schools in 2018-2019 and 11 schools in 
2019-2020) as a benchmark for the Uncommon model. For each component, we first compared the mean 
value for each of the four TurnNJ schools to the mean value for all non-TurnNJ Uncommon elementary 
and middle schools in New Jersey. We considered a TurnNJ school to have implemented the component 
with fidelity if the school mean equaled or exceeded the non-TurnNJ Uncommon mean. We then 
calculated the percentage of TurnNJ schools that met this standard for implementing the component with 
fidelity. 


We found that during the 2018-2019 school year, the three TumNJ schools operating during that time 
period and included in the impact analysis implemented five of six components of the Uncommon school 
model with fidelity (Table D.2). This finding suggests that the estimated impacts on students reflect the 
impacts of the TurnNJ project as intended. During the 2019-2020 school year, only two of the six 
components met our standards for implementation fidelity. However, in both years, a majority of the 
TurnNJ schools had mean values similar to the mean values of other Uncommon schools across all 
components (Tables D.3 and D.4). For each component, a majority of TurnNJ schools had a value within 
at least three-tenths of the Uncommon mean on measures using a 5-point scale. In addition, for most 
components, all TurnNJ schools had a value within the range of values for non-TurnNJ schools in the 
Uncommon network. (The results are similar when we include all four TurnNJ schools operating during 
the 2018-2019 and 2019-2020 school years.) 
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Table D.2. Percentage of TurnNJ schools implementing with fidelity, by key component and 


school year 


Key components of TurnNJ 
Yo aToto) Munreye(=1| 


2018-2019 


Percentage of 
schools 
implementing 
with fidelity? 


Met fidelity 
rtalast=ate) (° be 


VA EAIAN) 


Percentage of 
Tod aToXo) f= 
implementing 
with fidelity? 


Met fidelity 
talasts1ate) (° be 


College preparatory mission 67 Yes 33 No 
High academic and character 

standards 67 Yes 67 Yes 
Highly structured learning 

environment 33 No 33 No 
Longer school day and year 100 Yes 100 Yes 
Accountability and data-driven 

instruction 67 Yes 33 No 
Committed and talented leaders 

and teachers 67 Yes 33 No 


Source: Uncommon staff surveys administered in the 2018-2019 and 2019-2020 school years. School-level means 
were calculated as a simple average of staff survey responses. Staff response rates ranged from 90 to 94 
percent, so nonresponse weights were not used. 

@ TurnNJ school is implementing a key component of the Uncommon school model with fidelity if the school mean 

equals or exceed the Uncommon network mean. 


> Meeting the fidelity threshold indicates that at least 60 percent of schools are implementing the key component with 


fidelity. 
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Table D.3. Implementation of the Uncommon school model during the 2018-2019 school year, by 
school and key component 

Ua Texeyantanteyal TurnNJ TurnNJ TurnNJ 


Key components of TurnNJ network mean school A Xe) aTote) =) Yed sTote) i @3 
Ted ToL) Mu itere (=| (elite (=) be aT=x- 1a) mean mean 


College preparatory mission (single 

item, 1-5 scale) 4.6 (4.2-5.0) 4.6 4.8 4.5 
High academic and character standards 

(composite measure, 1—5 scale) 4.3 (3.8-4.8) 46 4.4 3.7 
Highly structured learning environment 

(composite measure, 1-5 scale) 4.1 (3.6-4.7) 3.8 4.7 3.6 
Accountability and data-driven 

instruction (single item, 1-5 scale) 4.3 (4.0-4.9) 4.3 4.4 3.9 
Committed and talented leaders and 

teachers (composite measure, 1-5 

scale) 4.4 (3.8-4.8) 4.5 4.6 3.8 


Source: Uncommon staff surveys administered in the 2018-2019 school year. School-level means were calculated 
as a simple average of staff survey responses. Staff response rates ranged from 90 to 94 percent, so 
nonresponse weights were not used. 

Note: The table does not include one component, longer school day and year, because school day length and 
year were consistent across all elementary schools in New Jersey and across all middle schools in New 
Jersey, including the TurnNJ schools. During the 2018-2019 school year, elementary school instructional 
time totaled approximately 1,710 hours, and middle school instructional time totaled approximately 1,777 
hours. 

4 Includes all elementary and middle schools operating in New Jersey as part of the Uncommon network during the 

2018-2019 school year. 
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Table D.4. Implementation of the Uncommon school model during the 2019-2020 school year, by 
school and key component 


Uncommon 
Key components of TurnNJ network mean | TurnNJ school | TurnNJ school | TurnNJ school 
Ted aYoLo) Mu itere (=| (range)? B mean C mean 
College preparatory mission (1-5 
scale) 4.6 (4.1-4.9) 4.5 49 4.2 
High academic and character 
standards (1—5 scale) 4.4 (3.9-4.8) 4.3 4.7 3.9 
Highly structured learning 
environment (1-5 scale) 4.2 (3.0-4.8) 3.9 4.5 4.0 
Accountability and data-driven 
instruction (1-5 scale) 4.4 (3.9-4.8) 4.5 4.7 3.8 
Committed and talented leaders and 
teachers (1—5 scale) 4.4 (4.2-4.7) 4.2 4.7 4.2 


Source: Uncommon staff surveys administered in the 2019-2020 school year. School-level means were calculated 
as a simple average of staff survey responses. Staff response rates ranged from 90 to 94 percent, so 
nonresponse weights were not used. 

Note: The table does not include one component, longer school day and year, because school day length and 
year were consistent across all elementary schools in New Jersey and across all middle schools in New 
Jersey, including the TurnNJ schools. During the 2019-2020 school year, elementary school instructional 
time totaled 9 hours per day before pandemic-related closures and 8.25 hours per day after pandemic- 
related closures, with a total school year length of 190 days. Middle school instructional time totaled 9 hours 
per day before pandemic-related closures and 8.5 hours per day after pandemic-related closures, with a 
total school year length of 185 days. 


4 Includes all elementary and middle schools operating in New Jersey as part of the Uncommon network during the 
2019-2020 school year. 
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